Gray release and traffic control method and device for large model service

Abstract

The embodiment of the application provides a gray release and traffic control method and device for a large model service, effectively monitors the state by innovatively designing a version evaluation system, and through index analysis and stability scoring. A traffic control mechanism is constructed, combined with routing decision and resource scheduling, to establish a reliable release strategy. Adaptive adjustment is introduced to ensure the smoothness of the release through state monitoring and rule optimization. The method effectively solves the deficiencies of traditional technology in version evaluation, traffic control and dynamic adjustment, and provides technical support for large model services.

Description

Method and apparatus for canary release and traffic control of large model services Technical Field

[0001] This application relates to the field of data processing, specifically to a method and apparatus for canary release and traffic control of a large model service. Background Technology

[0002] Existing methods for canary deployment of large-scale model services have significant shortcomings. Traditional systems perform poorly in version evaluation and traffic analysis, failing to effectively guarantee service stability and impacting deployment effectiveness.

[0003] Furthermore, existing technologies suffer from bottlenecks in routing decisions and resource scheduling. Most systems lack robust weight allocation mechanisms and scheduling strategies, resulting in inadequate load distribution.

[0004] The existing system has technical shortcomings in dynamic adjustment. It lacks in-depth analysis of its operational status, making it difficult to achieve flexible traffic control through strategy optimization, thus impacting service quality. Solving these problems is crucial for improving the service capabilities of large-scale models. Summary of the Invention

[0005] To address the problems in the existing technology, this application provides a method and apparatus for canary release and traffic control of large model services, which can effectively solve the shortcomings of traditional technologies in version evaluation, traffic control and dynamic adjustment, and provide technical support for large model services.

[0006] To solve at least one of the above problems, this application provides the following technical solution:

[0007] Firstly, this application provides a method for canary release and traffic control of a large model service, including:

[0008] Collect operational metrics data for the model service group, including request success rate, average inference latency, business conversion metrics, model output distribution characteristics, and traffic fluctuation patterns. Establish a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Construct a version status evaluation matrix based on the operational metrics data, calculate version stability score based on the version status evaluation matrix, and perform cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0009] Based on the version stability score, initial traffic weights are assigned to multiple versions within the model service group. A routing decision table containing version identifiers, weight values, and session persistence rules is generated. A computing resource scheduling strategy between versions is constructed. Inference nodes are dynamically allocated according to the computing resource scheduling strategy. The version selection probability is calculated to generate a canary release scheme containing version allocation sequences, resource configuration parameters, and adaptive adjustment rules. The abnormal traffic feature library is used to filter abnormal requests and limit traffic.

[0010] The canary release scheme is deployed to the inference service gateway, which monitors the version's running status and generates running logs containing changes in request volume, error rate statistics, performance fluctuation data, and resource utilization. Based on the running logs, the load distribution between versions is analyzed and resource rebalancing is triggered. The canary ratio is dynamically adjusted according to the adaptive adjustment rules. When the rollback condition is triggered, traffic is switched to the stable version and the routing decision table is updated, and version status change information is recorded.

[0011] Furthermore, it also includes: collecting real-time running data streams from the inference service gateway, extracting request processing result identifiers, inference calculation time, target task completion markers, prediction result probability distributions, and request arrival time series, segmenting and aggregating the running data streams according to fixed time windows, calculating request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns, and constructing a running indicator dataset;

[0012] The operational metric dataset is mapped to the version management system, a model service group structure is created, a globally unique identifier is assigned to each version, the model weight file path, inference container configuration information, and computing resource quota are written in, a version dependency graph is established based on the globally unique identifier, and an associated data structure containing version basic information and operational metrics is generated.

[0013] Furthermore, it also includes: grouping the operational indicator data according to version identifiers, extracting the indicator time series for each version, calculating the indicator mean, variance, trend slope, and mutation point location, constructing a state matrix containing multi-dimensional evaluation features, normalizing the state matrix, generating a version state evaluation matrix, and calculating the version stability score based on the version state evaluation matrix.

[0014] Request frequency, resource consumption, and time distribution features are extracted from traffic fluctuation patterns. Density clustering algorithm is used to group the features, and the center vector and boundary threshold of each cluster are calculated. The center vector and boundary threshold are combined to form an abnormal traffic feature library and a feature matching rule set is established.

[0015] Furthermore, it also includes: constructing a normalized weight vector based on the version stability score, smoothing the weight vector to eliminate weight mutations, generating a version priority sequence in combination with the version release time, calculating the initial traffic weight allocation ratio according to the version priority sequence, and establishing a session identifier and version mapping relationship;

[0016] The initial traffic weight allocation ratio is converted into a cumulative weight range, and a routing decision table containing version identifiers, range boundaries, and session persistence rules is generated. Based on the load status of computing nodes, an inter-version resource scheduling strategy is constructed, and the resource scheduling strategy is written into the scheduling constraint field of the routing decision table.

[0017] Furthermore, it also includes: obtaining the computing load level of each node based on the computing resource scheduling strategy, grouping and marking the inference nodes, the available computing power capacity and network bandwidth of the computing node group, dynamically allocating inference nodes for each version according to the available computing power capacity and network bandwidth, and generating a node allocation mapping table.

[0018] The node allocation mapping table is used to calculate the version selection probability, and a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules is constructed. The abnormal traffic feature library is imported into the traffic filtering module, and request filtering thresholds and rate limiting triggering conditions are set based on the abnormal traffic feature library to generate a traffic control rule set.

[0019] Furthermore, it also includes: parsing the canary release scheme into gateway configuration instructions, updating the routing rule table based on the gateway configuration instructions, starting the version running status collector, collecting request volume change curves, error rate time series, performance index fluctuations, resource usage levels, and generating running logs containing multi-dimensional monitoring data;

[0020] The operation logs are subjected to time-series analysis to calculate the differences in request distribution and resource utilization between versions, and a load balancing score index is constructed. The degree of resource allocation imbalance is judged based on the load balancing score index. When the imbalance exceeds a preset threshold, a resource rebalancing operation is triggered to update the node allocation scheme.

[0021] Furthermore, it also includes: analyzing the current canary release progress based on adaptive adjustment rules, calculating the performance index scores of each version, adjusting the version traffic allocation ratio according to the performance index scores, generating a new canary weight vector, and writing the canary weight vector into the routing decision table.

[0022] The monitoring of version operation metrics triggers a rollback decision. When a performance degradation metric is detected to exceed the threshold, the allocation of traffic to the new version is paused, all request traffic is switched to the stable version, the routing decision table configuration items are updated, and a version status change log is recorded, including the rollback reason, the switching time, and the scope of impact.

[0023] Secondly, this application provides a canary release and traffic control device for a large model service, comprising:

[0024] The service monitoring module is used to collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. It establishes a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Based on the operational indicator data, it constructs a version status evaluation matrix, calculates version stability score according to the version status evaluation matrix, and performs cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0025] The canary release module is used to allocate initial traffic weights to multiple versions within the model service group based on the version stability score, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability to generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0026] The traffic control module is used to distribute the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data and resource utilization. Based on the running logs, it analyzes the load distribution between versions and triggers resource rebalancing. It dynamically adjusts the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, it switches traffic to the stable version and updates the routing decision table, and records the version status change information.

[0027] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the gray-scale release and traffic control method for the large model service.

[0028] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the gray-scale release and traffic control method for the large model service.

[0029] Fifthly, this application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the gray-scale release and traffic control method for the large model service.

[0030] As described above, this application provides a method and apparatus for canary deployment and traffic control of large-scale model services. Through an innovative version evaluation system, it achieves effective status monitoring via indicator analysis and stability scoring. A traffic control mechanism is constructed, combining routing decisions and resource scheduling to establish a reliable deployment strategy. Adaptive adjustment is introduced, ensuring deployment stability through status monitoring and rule optimization. This method effectively addresses the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large-scale model services. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 is a flowchart illustrating the canary release and traffic control method for large model services in an embodiment of this application;

[0033] Figure 2 is a structural diagram of the grayscale release and traffic control device for large model services in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of the structure of the electronic device in the embodiment of this application.

[0035] Figure label:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The acquisition, storage, use, and processing of data in this application comply with relevant laws and regulations.

[0039] To address the problems existing in current technologies, this application provides a method and apparatus for canary deployment and traffic control of large-scale model services. Through an innovative version evaluation system, it achieves effective status monitoring via indicator analysis and stability scoring. A traffic control mechanism is constructed, combining routing decisions and resource scheduling to establish a reliable deployment strategy. Adaptive adjustment is introduced, ensuring stable deployment through status monitoring and rule optimization. This method effectively solves the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large-scale model services.

[0040] To effectively address the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, and to provide technical support for large model services, this application provides an embodiment of a canary release and traffic control method for large model services. Referring to Figure 1, the canary release and traffic control method for large model services specifically includes the following:

[0041] Step S101: Collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. Establish a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Construct a version status evaluation matrix based on the operational indicator data. Calculate the version stability score based on the version status evaluation matrix. Perform cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0042] This embodiment receives operational metric data streams, with data originating from request-level records of the inference service gateway and container-side resource observations. The data is aligned at a fixed time granularity, establishing synchronized segments with five fields: request success marker, inference time consumption, business completion marker, output probability vector, and arrival time series. Lost samples are linearly interpolated using the steady-state intervals of adjacent segments, and the interpolation markers are retained for subsequent weight reduction, avoiding misjudgments in unexpected situations.

[0043] After data cleaning, time window slices are constructed for the runtime metric dataset. Each version corresponds to a set of metric time series, and the moving average and variance of request success rate and average inference latency are calculated respectively. Business conversion metrics are given as the cumulative proportion of completions within the window. The output probability vector is binned to obtain the distribution morphology features, including dimensionless representations of skewness and kurtosis. For the arrival time series, frequency curves of traffic fluctuation patterns are obtained using fixed-step statistics, and the relative positions of peaks are recorded for subsequent clustering alignment.

[0044] Based on this, a state matrix is ​​constructed by grouping according to version identifiers. Rows in the matrix correspond to time windows, and columns correspond to indicator characteristics, including mean, variance, trend slope, and abrupt change locations. To eliminate dimensional differences, scaling and normalization are performed on the median and interquartile range for each column. Imputation marks for outlier segments are added to an additional weight channel to reduce the impact of that row in the scoring. To establish a unified evaluation domain, a stability scoring model, the "steady-state scorer," is introduced. Its output is a value between zero and one, used to quantify the stability of the version during the current observation period.

[0045] The stability score is calculated using a weighted linear form, and the formula is given below:

[0046] .

[0047] Where S is the version stability score, r is the normalized mean of request success rate, d is the normalized mean of average inference latency, v is the weighted sum of the variances of each indicator, c is the normalized value of mutation density within the window, and w1 to w4 are non-negative weights set by the strategy side and sum to one. The values ​​of each symbol are derived from the aforementioned state matrix and are entered into the summation after being corrected by interpolation weights. This form ensures that the score monotonically increases when the success rate increases, latency decreases, volatility weakens, and mutations are rare, which aligns with industry intuition regarding service stability.

[0048] To establish the model service group structure, a model service group containing multiple versions is created within the version management system. Each version includes the model weight file path, inference container configuration information, and computing resource quota, and a reference relationship is established between it and the runtime metric dataset using a global identifier. This association structure is directly invoked during subsequent traffic weight allocation and routing decision generation, ensuring consistency between metrics and version entities and preventing identifier drift.

[0049] To address traffic fluctuation patterns, feature vectors are constructed, including request frequency per unit time, resource utilization ratio, and peak location encoding. Density clustering is used to group these vectors, and the cluster centers and boundary thresholds are calculated to form an abnormal traffic feature library. To avoid misclassifying periodic peaks as anomalies, seasonal decomposition is performed on the frequency curves before clustering; only the residuals are included in the clustering space. Boundary thresholds are estimated using inflection points of local density, independent of absolute values.

[0050] After its generation, the abnormal traffic feature library is stored in the evaluation repository along with the version stability score. Subsequent steps read the stability score when allocating initial traffic weights and load abnormal features on the inference service gateway side for preliminary screening. The two form a constraint chain: the former determines the baseline weight that the version can bear, and the latter eliminates disruptive traffic and reduces its negative impact on the score, ensuring that the weight allocation is consistent with and traceable to the input of resource scheduling.

[0051] Step S102: Based on the version stability score, assign initial traffic weights to multiple versions within the model service group, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability, generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0052] This embodiment proceeds to the weight allocation phase after obtaining the version stability score. To avoid sudden changes in ratios due to short-term fluctuations, the score is first exponentially smoothed within a rolling window to obtain an effective score for allocation. A normalized weight vector is constructed based on the effective score. The weight vector is constrained by the version release time, with newer versions receiving a lower initial proportion for the same score, while older versions maintain a minimum proportion to prevent the service from experiencing excessive traffic during periods before full verification. An upper limit is imposed on the variation range of the weight vector to restrict unilateral changes within any period. The reason for the change and the calculation time are recorded as the basis for subsequent backtracking.

[0053] After the weights are determined, a routing decision table is generated. The routing decision table uses the version identifier as the primary key and contains weight values ​​and session persistence rules. Session persistence employs a mapping from session identifier to version identifier, with the mapping rules determined by both hash endpoints and the effective window. For already bound sessions, migration is triggered only when the version is marked as unavailable. Newly entering sessions select an endpoint based on the cumulative distribution of their current weights, with the endpoint range corresponding to the target version, forming a structure compatible with request-level probabilistic routing and session-level persistence. The routing decision table includes an effective time and an expiration time to avoid state fragmentation caused by concurrent updates.

[0054] Weights are merely the basis for traffic distribution at the entry point; the computational resource scheduling strategy determines the set of inference nodes available for each version. This embodiment uses node load, available memory, and network connectivity to form a ternary state. The node capacity is calculated based on the estimated throughput requirements of each version, resulting in an assignment matrix from version to node. To maintain balance among nodes, minimum-cost matching is employed, with the cost function weighted by the node's current utilization and historical jitter rate. Assignment results are presented as node groups, labeled as primary and backup groups. The primary group handles regular requests, while the backup group takes over when the primary group's load exceeds a threshold or node health declines. This strategy is periodically recalculated with updates to the runtime logs, but changes to the routing decision table are applied with a delay to ensure consistency.

[0055] After node assignments are determined, the version selection probability is calculated. This probability is jointly determined by the entry weight and the available computing power ratio of the assignment matrix. The entry weight reflects the intent, and the computing power ratio reflects the supply. The two are multiplied and normalized to obtain the final selection probability. A version allocation sequence is generated based on the selection probability, which is used as a behavioral approximation in high-concurrency scenarios. Low-overhead landing points can be achieved in a stateless gateway using a pre-generated array. Resource configuration parameters are generated synchronously, including the concurrency limit, connection pool capacity, and cold start warm-up quantity for each version on its node group, serving as the execution boundary on the inference container side.

[0056] The adaptive adjustment rules are solidified into executable entries at this stage. The rules use deviations in runtime log metrics as triggers, defining the adjustment step size and cooldown time. For example, if the average inference latency of the target version is below the baseline for a period of time and the error rate does not increase, then the weight of that version is slightly increased after the cooldown period ends; conversely, if the error rate increases and is accompanied by resource utilization saturation, then the entry weight is reduced and a resource rebalancing request is triggered. These rule entries are written into the canary release plan for direct evaluation and execution in subsequent stages.

[0057] An abnormal traffic feature library is enabled as a pre-filter at the ingress side. When a request enters the gateway, it is matched against the center and boundary thresholds in the feature vector library. If a match is found, rate limiting or degradation is implemented. Rate limiting actions include dropping, queuing, or redirecting to a lower-weight older version. The rate limiting ratio and queuing limit are explicitly given by the rule set. To reduce the risk of false positives, a tentative allowance ratio is set for requests near the boundary, and these requests are marked in the operation logs for differentiation during backtracking.

[0058] The above processes are summarized into a canary release scheme, which includes a version allocation sequence, resource configuration parameters, and adaptive adjustment rules, along with a routing decision table and version numbers for node assignment results. Downstream, the scheme is parsed into gateway update commands and node-side container parameters, aligning the entry point and the computing side's effective window to avoid temporary inconsistencies where weights have been adjusted but computing power is not yet in place. Subsequent steps involve status monitoring and rebalancing, and the recorded logs are returned to evaluate the adjustment effects and correct weights and assignments for the next cycle.

[0059] Step S103: Deploy the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Analyze the load distribution between versions based on the running logs and trigger resource rebalancing. Dynamically adjust the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, switch traffic to the stable version and update the routing decision table, and record the version status change information.

[0060] This embodiment proceeds to the distribution and monitoring phase after the canary release plan is signed. The plan primarily consists of version allocation sequences, resource configuration parameters, and adaptive adjustment rules, supplemented by a routing decision table and version numbers of node assignment results. Upon receiving the data, the gateway performs a two-phase update: In the pre-loading phase, a new cumulative weight range and session persistence mapping cache are generated in an independent memory area, while connection probes are established with the node group; in the switching phase, the routing pointer is atomically replaced with the new table within a predetermined effective window, and the old table is placed in a read-only buffer, retaining short-term replays to cover pending requests. A one-time sequence number is appended to the switching operation to prevent concurrent writes from compromising consistency.

[0061] Monitoring begins immediately upon switchover completion, with the collector aggregating four types of data in a windowed manner. Request volume changes are recorded as the request count and instantaneous peak position per unit time for each version; error rate statistics are categorized by status code and exception type; performance fluctuation data originates from inference latency percentile information reported by containers; and resource utilization is comprised of node-side calculations and memory usage observations. Each data point is used to establish a time-series entry with the version identifier and assigned node group as keys, and interpolation markers and clock offset fields are written for subsequent alignment and reliability assessment. To minimize the impact of monitoring on the path, sampling and batch submission are used for data collection, avoiding complex calculations on hot paths.

[0062] Load analysis is performed in the computation unit bypassed by the gateway. After aligning the runtime logs, the deviation in request percentage between versions and the deviation in node group resource utilization are calculated to form a load balancing score. If the score exceeds the threshold and persists for multiple windows, a resource rebalancing suggestion is generated. The suggestion includes the timing for expanding the node group of the target version and promoting the backup group to the primary group. The rebalancing does not take effect immediately, but enters the scheduling queue. In the next safe window, the assignment matrix is ​​fine-tuned and bound to the next update of the routing decision table to avoid misalignment between the entry point and the provisioning.

[0063] Adaptive adjustment rules are evaluated within the same analysis workflow. Rules are based on metric deviation and cooldown time; when these conditions are met, grayscale adjustment entries are generated. These adjustments employ triple constraints—small step size, rate limit, and upper limit—and are written into the differential representation of the weight vector to be applied, ensuring continuity and traceability. If logs show an increased abnormal traffic filtering hit rate and a simultaneous increase in the error rate of the target version, the adjustment logic will prioritize reducing the target version's ingress weight and request additional anomaly feature verification to avoid being dragged down by traffic from a single source.

[0064] Rollback decisions are triggered by independent threshold conditions and do not share a cooldown time with regular adjustments. When any one of the following three strong trigger conditions is met simultaneously: an upward surge in error rate, a deterioration in performance percentile, or abnormal resource utilization, sessions newly added to the problematic version are immediately frozen, and all new requests are switched to the stable version. Session persistence mapping performs a soft interruption on bound sessions, allowing natural convergence at the end of the window; if necessary, a hard switch is performed on long sessions and a special flag is added. After the rollback operation is complete, the gateway generates a new routing decision table version, and the old version is archived and bound to the trigger reason.

[0065] To ensure operational traceability, version status change information is synchronously recorded after each switch, adjustment, and rollback. Recorded items include version identifier, change type, differential weight vector, node assignment differential, trigger metric summary, and effective time. This information is also written to the header of the runtime log, forming a closed-loop link from entry decision to resource allocation. In subsequent cycles, the scorer and assigner read the causal fragment of the most recent change from this link to adjust the smoothing parameters and thresholds for the next round, enabling the system to gradually converge without oscillations.

[0066] During a typical incremental gray-scale rollout phase, the new version's weight was slightly increased across multiple windows. Load analysis showed that the master group utilization was close to but not exceeding limits, and the rebalancing suggestion was accepted to expand the master group nodes. Subsequently, at a certain period, the anomaly hit rate increased, accompanied by a rise in the error rate. The adaptive rules paused their upward adjustment and triggered a rollback decision. The system completed the switchover within one window, with the stable version taking over the traffic. All relevant changes and their causes were fully recorded for subsequent review and policy correction.

[0067] As described above, the canary release and traffic control method for large model services provided in this application can effectively monitor the status through an innovative version evaluation system, indicator analysis, and stability scoring. It constructs a traffic control mechanism, combining routing decisions and resource scheduling to establish a reliable release strategy. Adaptive adjustment is introduced, and the stability of the release is ensured through status monitoring and rule optimization. This method effectively solves the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large model services.

[0068] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0069] Step S201: Collect real-time running data stream from the inference service gateway, extract request processing result identifier, inference calculation time, target task completion mark, prediction result probability distribution, and request arrival time series, segment and aggregate the running data stream according to a fixed time window, calculate request success rate, average inference latency, business conversion index, model output distribution characteristics, and traffic fluctuation pattern, and construct a running index dataset;

[0070] Step S202: Map the running metric dataset to the version management system, create a model service group structure, assign a globally unique identifier to each version, write the model weight file path, inference container configuration information, and computing resource quota, establish a version dependency graph based on the globally unique identifier, and generate an associated data structure containing version basic information and running metrics.

[0071] This embodiment constructs a lightweight data extraction channel on the inference service gateway side to continuously collect real-time running data streams. The channel identifies and parses request-level metadata and container-side observations, extracting five types of elements: request processing result identifier, inference computation time, target task completion marker, prediction result probability distribution, and request arrival time series. To ensure comparability, the acquisition end aligns the data according to fixed time windows, with the window boundaries synchronized with the effective time of the gateway routing decision table. For long requests spanning multiple windows, the window corresponding to the midpoint of the response time is used for accounting, and an occupancy marker is written to adjacent windows to avoid duplicate statistics.

[0072] During the computation phase within the window, request success rate and business conversion metrics are obtained based on the request processing result identifier and the target task completion marker. Both metrics use valid requests within the window as the denominator, excluding rate-limited samples that exhibit abnormal characteristics. Inference computation time is processed by removing extrema and truncating quantiles to obtain a robust estimate of average inference latency. The probability distribution of prediction results is binned according to fixed boundaries, and the morphological characteristics of the output probability vector are represented by skewness, kurtosis, and tail proportion, constituting the model output distribution characteristics. The request arrival time series is counted at fixed steps to form a frequency curve, and the amplitude and phase of the periodic components are further calculated as the core description of traffic fluctuation patterns. These metrics, along with the window identifier and version identifier, are written into the runtime metric dataset, retaining the original counts and processing markers for verification in subsequent evaluation stages.

[0073] To improve data quality, consistency checks are performed on the runtime metrics dataset before writing. These checks include timestamp monotonicity, session-level deduplication, and cross-validation of node-side resource observations. If a request success rate and error count are found to be inconsistent, the window is marked as low confidence and its weight in the stability score is reduced. For samples with missing probability distribution fields, imputation is not performed; instead, the request is removed from the output distribution feature statistics to prevent the introduction of spurious peaks. The cleaned dataset is temporarily stored in the storage layer using an in-memory columnar structure for easy slicing by version and time.

[0074] The runtime metric dataset is then mapped to the version management system, creating a model service group structure containing multiple versions. The mapping process uses the service group name and version source record as anchors, assigning a globally unique identifier to each version. This identifier matches the version identifier in the routing decision table, ensuring consistency between the entry-side and evaluation-side entities. Under each version entity, three basic configurations are written: model weight file path, inference container configuration information, and computing resource quota. A configuration version number and effective time pair is established, forming a traceable configuration timeline.

[0075] A version dependency graph is constructed based on globally unique identifiers. Dependency edges represent inheritance chains of source versions, hotfix branches, or model weights, with compatibility descriptions and parameter difference summaries annotated on the edges. This graph is used for subsequent canary release strategy constraints, such as limiting session persistence migration across incompatible branches, or setting stricter weight change caps for new versions with longer inheritance chains. The graph is checked for acyclicity during updates to prevent the formation of unexplainable upgrade paths.

[0076] The final result is a relational data structure containing basic version information and operational metrics. This structure uses the version identifier as the primary key, connects to the time-series index of the operational metrics dataset, and includes configuration timelines and dependency edge references. Subsequent steps, such as constructing the version status evaluation matrix and calculating version stability scores, directly read the grouped metrics and corresponding configuration contexts from this structure, ensuring that the scores are consistent with the effective configuration at the time. The relational data structure also provides a read-only view to the routing and resource scheduling modules, ensuring that entry point weight allocation, node assignment, and abnormal traffic filtering use data from the same source, reducing decision-making biases caused by inconsistencies between multiple sources.

[0077] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0078] Step S301: Group the running indicator data according to the version identifier, extract the indicator time series of each version, calculate the indicator mean, variance, trend slope, and mutation point location, construct a state matrix containing multi-dimensional evaluation features, normalize the state matrix to generate a version state evaluation matrix, and calculate the version stability score based on the version state evaluation matrix.

[0079] Step S302: Extract request frequency, resource consumption, and time distribution features from the traffic fluctuation pattern, group the features using a density clustering algorithm, calculate the center vector and boundary threshold of each cluster, combine the center vector and boundary threshold to form an abnormal traffic feature library, and establish a feature matching rule set.

[0080] This embodiment uses the time index in the associated data structure to group the operational indicator data by version identifier, obtaining a windowed indicator sequence for each version. To suppress the influence of abnormal windows, a weight correction is first applied to the sequence based on low-confidence markers. The weights are reduced but not eliminated, thus limiting the influence of extreme samples in the statistics. The moving average and variance are calculated for request success rate, average inference latency, business conversion indicators, and model output distribution characteristics, respectively. The trend slope is estimated using the weighted least squares method within a fixed-length window. The location of abrupt change points is identified using a two-sided cumulative sum statistic, and the location information and confidence weights are output for subsequent aggregation.

[0081] Within a single version, a state matrix with multi-dimensional evaluation features is constructed. Rows in the matrix correspond to time windows, and columns contain four types of sub-vectors: mean, variance, trend slope, and mutation density for each indicator. Two modifiers from the data acquisition end—clock offset and interpolation markers—are also included as confidence factors for that row. To eliminate biases caused by different dimensions, the matrix is ​​robustly normalized column-wise using median shift and interquartile range scaling. Rows with lower confidence levels maintain smaller row weights after normalization and do not participate in strong threshold condition evaluation. Multiple version state matrices are stacked along the version dimension to obtain a horizontally comparable version state evaluation matrix.

[0082] Version stability scores are derived from the evaluation matrix. To maintain interpretability, a "steady-state scorer" is constructed, denoted as r for the normalized mean of request success rate, d for the normalized mean of average inference latency, v for the weighted sum of variances of multiple metrics, and c for the mutation density. The score is given by a weighted linear formula, with weights derived from the strategy configuration and fixed within the scoring period. The score is first calculated as a time-weighted mean within a version, then mapped across versions to obtain a range of zero to one; higher values ​​indicate more stable operation. The obtained score, along with the timestamp and configuration version number, is written back to the evaluation repository for direct reference during the weight allocation phase.

[0083] To address traffic fluctuation patterns, this embodiment extracts three types of features related to request rhythm from the operational metric dataset. Request frequency is represented by a spectral amplitude vector composed of fixed-step counts; resource usage is calculated using the mean and high quantile pairs of the node group's computation and memory utilization within a window; temporal distribution features are encoded by the relative position of peak occurrences, day-night ratio, and periodic intensity. These three types of sub-features are concatenated into a standardized feature vector, and windows with the same routing decision table version number are aligned to ensure semantic consistency with the ingress policy.

[0084] The density clustering algorithm operates within the aforementioned feature space, employing variable bandwidth kernel density estimation to determine the neighborhood radius, thus avoiding class confusion under a uniform scale. After clustering, the center vector of each cluster is calculated, and a boundary threshold is estimated. The threshold is determined jointly by the inflection point of local density and the Mahalanobis distance quantile within the cluster, constraining the overall shape while retaining a buffer for a small number of tail samples. Samples in the critical region are labeled with soft tags as a basis for tentative release in subsequent flow control rules.

[0085] The abnormal traffic feature library consists of cluster center vectors and boundary threshold pairs, along with version and node group context, used to distinguish between requests originating from natural peaks and unexpected surges during the gate screening phase. To enable fast matching at the gateway, a feature matching rule set is established. Rules are expressed using linear projection and threshold conditions, without relying on complex inference. Requests that match the rules are marked and enter the rate limiting and degradation branches; requests that do not match are routed according to their normal weights. Both the rule set and the feature library have effective and expiration times, and are replaced in a rolling window manner during version updates to avoid old rules misleading new allocation strategies.

[0086] The aforementioned evaluation matrix and abnormal traffic feature database are simultaneously incorporated into the subsequent weight allocation and node assignment processes after generation. The former serves as a stability basis for initial weight calculation, while the latter forms a pre-constraint at the traffic entry point. Both share the same time baseline and version identifier, ensuring a clear causal chain from scoring to filtering, facilitating the identification of the common context for weight changes and rate limiting triggers during subsequent backtracking.

[0087] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0088] Step S401: Construct a normalized weight vector based on the version stability score, smooth the weight vector to eliminate weight mutations, generate a version priority sequence in combination with the version release time, calculate the initial traffic weight allocation ratio according to the version priority sequence, and establish a session identifier and version mapping relationship.

[0089] Step S402: Convert the initial traffic weight allocation ratio into a cumulative weight range, generate a routing decision table containing version identifier, range boundary, and session persistence rules, construct an inter-version resource scheduling strategy based on the compute node load status, and write the resource scheduling strategy into the scheduling constraint field of the routing decision table.

[0090] This embodiment uses version stability scores as input and first constructs a normalized weight vector within the scoring period. The scores are exponentially smoothed, with the smoothing coefficient set according to the window length to avoid ratio fluctuations caused by short-term anomalies. An initial vector is obtained by summing and normalizing the smoothed version scores, and then a release time constraint is superimposed: if two versions have similar scores, the version released earlier receives a higher initial proportion, while the newer version is limited to a minimum and maximum range. To suppress drastic changes within a single period, a threshold condition for the magnitude of change is applied to the weights of adjacent periods, and the difference and triggering reasons are recorded for subsequent backtracking.

[0091] A version priority sequence is determined over a continuous weight domain. Priority is determined by two factors: the ranking of smooth scores and the order of release time. These factors are merged lexicographically to generate a sequence free of ties. An initial traffic weight allocation ratio is calculated based on this version priority sequence, ensuring that versions with higher priority receive a higher proportion when scores are equal, while those with lower priority maintain a lower but non-zero proportion until the sequence stabilizes, thus creating a gradual load. The resulting ratio vector is projected using least-squares projection onto the ratio from the previous period to limit the total variation and form an executable initial weight.

[0092] Sessions maintain a stable mapping between session identifiers and versions. This embodiment employs a replayable hashing method, mapping session identifiers to range indices, and then to version identifiers. To avoid large-scale migrations caused by weight changes, a consistent hashing ring and buffer slots are introduced. When weights are slightly adjusted, only sessions near the boundary migrate; when a version is marked as unavailable, the mapping is redirected, and the hash point shifts to an adjacent available version in the priority sequence. Bound sessions remain unchanged throughout their lifecycle unless a rollback or failure occurs.

[0093] Based on the initial traffic weight allocation ratio, a cumulative weight interval is constructed. The weight vector is summed using prefix summation to obtain the interval boundary, with the upper bound corresponding to the total sum and the lower bound being the upper bound of the previous version. A routing decision table is generated, with entries containing version identifiers, interval boundaries, and session persistence rules. The session persistence rules describe two scenarios: new sessions select a version based on the current interval, and old sessions directly reuse historical versions based on the mapping cache. The decision table includes an effective time and an expiration time, managed with monotonically increasing version numbers, supporting atomic switching and replay.

[0094] The inter-version resource scheduling strategy is constructed based on the compute node load status. Node status includes three dimensions: compute utilization, memory availability, and link saturation. The load capacity is estimated based on the version's predicted request intensity, forming a feasible set of nodes from version to node. The strategy calculation adopts cost minimization matching. The cost function combines current utilization and historical jitter, favoring the selection of stable nodes into the primary load set. Each version obtains two types of node sets: a primary group and a backup group. The primary group handles normal traffic, while the backup group takes over when the primary group's load approaches a threshold or its health declines. The strategy also generates a concurrency limit and a cold start quota to constrain the instantaneous expansion of the container layer.

[0095] To ensure that ingress routing aligns with supply capacity, resource scheduling policies are written into the scheduling constraint field of the routing decision table. This field records the available node group identifiers, concurrency limits, and switching conditions for each version. After selecting a landing point, the gateway must also verify the remaining capacity of the primary group for the corresponding version. If the requirements are not met, the request is redirected to the backup group or an adjacent version according to the constraints. Scheduling constraints and interval boundaries work together to prevent the release of requests exceeding supply capacity at the ingress level, reducing queuing and timeout risks.

[0096] After the routing decision table is generated, it enters the pre-release verification stage. Verification includes interval coverage, session mapping stability, and constraint consistency. Coverage requires no gaps or overlaps in the intervals; mapping stability is calculated by sampling sessions to determine the migration ratio, and if it exceeds a threshold, the weight change range is tightened; constraint consistency verifies that the primary and backup groups of the versions do not conflict globally. If the verification passes, it proceeds to the downstream deployment and monitoring stage; if it fails, it returns to this step to adjust weights or policy parameters until the execution conditions are met.

[0097] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0098] Step S501: Obtain the computing load level of each node based on the computing resource scheduling strategy, group and mark the inference nodes, calculate the available computing power capacity and network bandwidth of the computing node group, dynamically allocate inference nodes for each version according to the available computing power capacity and network bandwidth, and generate a node allocation mapping table.

[0099] Step S502: Calculate the version selection probability using the node allocation mapping table, construct a gray-scale release scheme that includes version allocation sequence, resource configuration parameters, and adaptive adjustment rules, import the abnormal traffic feature library into the traffic filtering module, set request filtering thresholds and rate limiting triggering conditions based on the abnormal traffic feature library, and generate a traffic control rule set.

[0100] This embodiment reads node status snapshots from the computing resource scheduling strategy to obtain the computing load level of each node in the current window. The load is composed of three observations: core utilization, memory usage ratio, and link congestion. The observation values ​​are robustly statistically analyzed within the window to suppress spikes. Based on the hardware isomorphism of the nodes and the adjacency of the network topology, the inference nodes are grouped and labeled to form several node groups. The latency difference and bandwidth fluctuation within each group are small, making them convenient as independent units for scheduling. The available computing power capacity of each node group is calculated as the weighted sum of the number of available cores in the group, with the weights inversely proportional to the historical jitter rate. The network bandwidth is represented by the product of the minimum available bandwidth within the group and the link stability to ensure that the feasibility boundary of cross-device communication is clearly defined.

[0101] After obtaining the available computing power and network bandwidth of the node groups, inference nodes are dynamically allocated for each version. Allocation is based on the version's expected request intensity and average computational cost per request. First, the version's capacity score in each node group is calculated. This capacity score serves as the joint lower limit of the computing power and bandwidth thresholds, ensuring that allocation does not exceed any bottleneck. The primary capacity group is selected from highest to lowest score, and several backup capacity groups are chosen within a range where the score decreases but still meets the minimum availability requirements. Concurrency limits, connection pool quotas, and cold start warm-up quantities are generated for the primary and backup groups respectively, serving as hard constraints at the container layer. These decisions are summarized into a node allocation mapping table. Each entry uses the version identifier as the key, mapping to the group identifier and quota parameters of the primary and backup groups, and includes an effective window and verification fingerprint to ensure consistency in subsequent distributions.

[0102] The version selection probability on the ingress side is corrected using a node allocation mapping table. The initial probability is derived from the weight range of the routing decision table, and the correction coefficient is the product of the available computing power ratio of each version's primary bearer group and the network bandwidth margin, both reflecting the supply capacity. The initial probability is multiplied by the correction coefficient and then normalized within the version set to obtain the version selection probability for the current window. Based on this probability, a version allocation sequence is generated. The sequence is a fixed-length integer array where version identifiers repeat proportionally, facilitating constant-time endpoint completion by the gateway on high-concurrency paths.

[0103] Resource configuration parameters are directly derived from the mapping table, including the concurrency limit, connection pool size, cold start warm-up quantity, and switching threshold for each version in the primary and backup groups. To control the adjustment pace, adaptive adjustment rules are constructed. These rules are triggered by latency percentiles, error rates, and utilization deviations in the runtime logs, specifying the weight fine-tuning step size and cooldown time. Rules do not cross container quota boundaries to avoid queue bloat caused by increasing entry weights when supply has not been expanded. Rule entries include a fallback path; if two consecutive negative signals are triggered, the most recent increase is automatically revoked.

[0104] The abnormal traffic feature library is imported into the traffic filtering module at this stage. The module constructs a feature vector for each request at the gateway ingress and matches it against the center and boundary thresholds in the feature library. Based on the matching results, request filtering thresholds and rate limiting trigger conditions are set, forming a traffic control rule set. The rule set specifies three types of actions: dropping, queuing, and downgrading to a stable version, and provides their respective upper limit ratios and timeout policies. Requests near the boundary are tentatively allowed into the regular route at a certain percentage, and are simultaneously marked in the operation log for subsequent threshold correction in the scoring process.

[0105] The canary release scheme consists of three parts: version allocation sequence, resource configuration parameters, and adaptive adjustment rules. It also includes version numbers for the node allocation mapping table and routing decision table for verifying the effective order. After the scheme is output downstream, the gateway and nodes execute it within the same effective window, ensuring that the entry probability and supply capacity remain consistent, reducing fluctuations caused by short-term mismatches. The operation logs will collect the rate limiting hit rate and the backup group takeover frequency, serving as the basis for recalculating the mapping table and rule set in the next cycle.

[0106] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0107] Step S601: Parse the canary release scheme into gateway configuration instructions, update the routing rule table based on the gateway configuration instructions, start the version running status collector, collect request volume change curves, error rate time series, performance index fluctuations, resource usage levels, and generate running logs containing multi-dimensional monitoring data;

[0108] Step S602: Perform time-series analysis on the running logs, calculate the differences in request distribution and resource utilization between versions, construct a load balancing score index, determine the degree of resource allocation imbalance based on the load balancing score index, trigger resource rebalancing operation when the preset threshold is exceeded, and update the node allocation scheme.

[0109] This embodiment receives the canary release scheme output from the previous stage and parses the version allocation sequence, resource configuration parameters, and adaptive adjustment rules into executable gateway configuration instructions. The gateway side employs a double-buffered update mechanism. First, a new routing rule table is generated in the backup buffer, including cumulative weight ranges, session persistence mapping caches, and verification hooks for node group constraints. After consistency verification, an atomic switch is performed within a predetermined effective window, and the old table is converted to read-only mode to overwrite pending requests. To prevent sudden fluctuations, the switchover action includes a minimum retention time, prohibiting secondary updates within this window.

[0110] Once the routing rule table is successfully updated, the version runtime status collector is launched. The collector is mounted on both the gateway and node sides, aggregating four types of monitoring data in a unified window. The request volume change curve consists of the request count per unit time for each version and the peak time position; the error rate time series is bucketed by error type and status code; performance index fluctuations consist of inference latency percentiles and queue waiting time reported from the container side; resource utilization levels include compute utilization, GPU memory usage ratio, and link bandwidth margin. Each data entry carries a version identifier, node group identifier, and effective route version number to ensure traceability for subsequent alignment. Missing segments are marked with sampling gaps but not imputed; their reliability is downgraded to avoid affecting threshold condition judgments.

[0111] Before writing the runtime logs, a fast consistency check is performed. The check items include the closure relationship between request and error counts, whether the session persistence migration ratio is within the change limit set in the previous period, and whether the node group concurrency has exceeded the concurrency limit in the resource configuration parameters. If any check fails, an exception is marked and a downsampling strategy is triggered to reduce the impact of that window's data on the adjustment rules. Logs that pass the check are persisted in a columnar structure and retain the window-level hash fingerprint for subsequent replay.

[0112] Based on the aforementioned runtime logs, time-series analysis was conducted, focusing on calculating the differences in request distribution and resource utilization between versions. Request distribution differences were represented by the weighted norm of the difference between the request percentage of each version within the current window and the target percentage of the routing. The weight was inversely proportional to the version stability score, preventing the amplification of low-reliability windows. Resource utilization deviation was represented by the maximum value of the standardized distances between three parameters: node group computation utilization, memory usage ratio, and bandwidth margin, reflecting the tightest constraints on the supply side. Both types of differences were incorporated into the load balancing scoring index; a higher score indicated a greater inconsistency between ingress traffic distribution and supply capacity.

[0113] The degree of resource allocation imbalance is determined based on load balancing scoring metrics. Windows with scores consistently exceeding a preset threshold are marked as requiring rebalancing, and a rebalancing request is generated, including the versions that need to be expanded or contracted and the corresponding candidate node groups. The rebalancing operation is performed in the next safe window. The scheduler fine-tunes the node allocation scheme, prioritizing the reallocation of concurrent quotas within the same group. If there is no spare capacity within a group, a backup group is introduced for improvement. To ensure ingress consistency, updates to the node allocation scheme are tied to the next version switch of the routing rule table, preventing ingress probability adjustments from taking effect before supply adjustments.

[0114] In addition to rebalancing, adaptive adjustment rules are also evaluated within the same analysis period. When the target version exhibits low latency and stable error rate across multiple windows, while the load balancing score is low, a slightly increased weight entry is generated; conversely, if the error rate rises and is accompanied by resource saturation, a combined entry of weight reduction and backup group expansion is generated. All entries record a summary of triggering metrics and cooldown time; entries that do not meet the cooldown conditions are not included in the execution queue to prevent frequent fluctuations.

[0115] After the node allocation scheme is updated, the new primary / standby group identifier, concurrency limit, and switchover threshold are written back to the resource configuration parameters and distributed along with the next version of the routing rule table. The operation logs then enter a new round of collection and analysis, forming a closed loop of "distribution—observation—evaluation—rebalancing." If strong conditions requiring a rollback trigger occur within the closed loop, the gateway directly performs a switch to the stable version and records the change along with the relevant score for subsequent review and policy revision.

[0116] In one embodiment of the canary release and traffic control method for large model services in this application, the following may also be included:

[0117] Step S701: Analyze the current canary release progress based on the adaptive adjustment rules, calculate the performance index score of each version, adjust the version traffic allocation ratio according to the performance index score, generate a new canary weight vector, and write the canary weight vector into the routing decision table.

[0118] Step S702: Monitor version operation metrics to trigger rollback determination. When the performance degradation metric is detected to exceed the threshold, pause the allocation of traffic to the new version, switch all request traffic to the stable version, update the routing decision table configuration items, and record the version status change log including the rollback reason, the switching time point, and the scope of impact.

[0119] This embodiment, assuming stable log collection, enters the adaptive adjustment phase. Based on the adaptive adjustment rules, the current canary release progress is analyzed, extracting three indicators for each version within the most recent several windows: latency quantile, error rate, and resource utilization deviation. These are used to construct an observation vector for scoring. To mitigate the interference of occasional spikes, the observation vector is time-weighted, with higher weights for more recent windows, and lower weights for windows marked as low confidence. Performance index scores are calculated based on this vector. The scoring model uses a weighted linear form; lower latency, lower error rate, and smaller deviation result in a higher score. The score is limited to the interval between zero and one, with an attached confidence interval width to constrain the subsequent weight changes.

[0120] After obtaining the performance metric scores, weight adjustment entries are generated according to the rules. Entries are divided into upward and downward adjustments, each requiring different trigger combinations and cooldown times. For example, an upward adjustment entry must simultaneously meet low latency, stable error rate, and low load balancing score; a downward adjustment entry can be triggered when either an error rate increase or latency degradation occurs and resource utilization approaches its upper limit. For each version, a suggested traffic allocation ratio increment is calculated. This increment is added to the previous period's grayscale weight vector to obtain a new grayscale weight vector, and normalization and variation constraints are applied to ensure that the single-period change does not exceed the set upper limit and that a minimum proportion of stable versions is maintained.

[0121] The new grayscale weight vector is written to the pending-application area of ​​the routing decision table. The write action carries the effective window and version number. The gateway atomically switches when it reaches the window boundary, and the old weight enters a read-only retention period to prevent in-flight requests from being affected. Session persistence mapping uses a consistent hashing buffer mechanism to ensure that session migrations caused by weight changes are limited to the vicinity of the interval boundary. If the migration ratio exceeds a threshold, an update is delayed and the reason for the anomaly is recorded. After the write is complete, the weight vector and the trigger entry summary are archived together for later backtracking.

[0122] The rollback determination is performed separately from the aforementioned minor adjustments, employing a more stringent set of strong trigger conditions. The system continuously monitors version performance metrics, and enters the determination process when performance degradation metrics exceed thresholds. Degradation metrics include three items: error rate exceeding the threshold, a sharp drop in latency percentiles, and abnormal resource utilization. A rollback is determined if any one of these strong triggers is triggered. Upon determination, the entry weight of the new version is immediately frozen, its range in the routing decision table is reduced to zero, and all request traffic is switched to the stable version. To reduce session interruptions, bound sessions are soft-exited first, naturally converging at the end of the window; a maximum duration is set for long sessions, and a hard switch is performed if this is exceeded, with a special flag indicating this.

[0123] After the rollback is completed, the configuration items in the routing decision table are updated synchronously, generating a new table version number. Metadata related to the rollback is compiled into a version status change log, including a summary of the rollback reason metrics, the switchover time, the scope of impact, and a list of involved node groups, along with a snapshot of the grayscale weight vector at that time. This log is written to the audit chain and subsequently used at the policy layer to adjust weight caps, extend cooldown times, or modify boundary thresholds in the abnormal traffic feature database. If resource constraints occur in the stable version after the rollback, the scheduler simultaneously triggers a resource rebalancing request, expanding capacity in the order of primary and backup groups to ensure service continuity.

[0124] To effectively address the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, and to provide technical support for large model services, this application provides an embodiment of a canary release and traffic control device for large model services, which implements all or part of the canary release and traffic control method for the large model service. Referring to Figure 2, the canary release and traffic control device for large model services specifically includes the following components:

[0125] Service monitoring module 10 is used to collect operational indicator data of model service groups, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. It establishes a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Based on the operational indicator data, it constructs a version status evaluation matrix, calculates version stability score according to the version status evaluation matrix, and performs cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0126] The canary release module 20 is used to allocate initial traffic weights to multiple versions within the model service group based on the version stability score, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability to generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0127] The traffic control module 30 is used to distribute the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data and resource utilization. Based on the running logs, it analyzes the load distribution between versions and triggers resource rebalancing. It dynamically adjusts the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, it switches the traffic to the stable version and updates the routing decision table, and records the version status change information.

[0128] As described above, the canary release and traffic control device for large model services provided in this application embodiment can effectively monitor the status through an innovative version evaluation system, indicator analysis, and stability scoring. It constructs a traffic control mechanism, combining routing decisions and resource scheduling to establish a reliable release strategy. Adaptive adjustment is introduced, and the stability of the release is ensured through status monitoring and rule optimization. This method effectively solves the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large model services.

[0129] From a hardware perspective, in order to effectively address the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, and to provide technical support for large-scale model services, this application provides an embodiment of an electronic device for implementing all or part of the canary release and traffic control method for the large-scale model service. The electronic device specifically includes the following components:

[0130] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to implement information transmission between the large-model service canary release and traffic control device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the large-model service canary release and traffic control method and the large-model service canary release and traffic control device in the embodiments, the content of which is incorporated herein, and repeated details will not be repeated.

[0131] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0132] In practical applications, the canary release and traffic control methods for large-scale model services can be partially executed on the electronic device side as described above, or all operations can be completed on the client device. The choice can be made based on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed on the client device, the client device may further include a processor.

[0133] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0134] Figure 3 is a schematic block diagram of the system configuration of an electronic device 9600 according to an embodiment of this application. As shown in Figure 3, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that Figure 3 is exemplary; other types of structures may also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0135] In one embodiment, the canary release and traffic control methods for large model services can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following controls:

[0136] Step S101: Collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. Establish a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Construct a version status evaluation matrix based on the operational indicator data. Calculate the version stability score based on the version status evaluation matrix. Perform cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0137] Step S102: Based on the version stability score, assign initial traffic weights to multiple versions within the model service group, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability, generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0138] Step S103: Deploy the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Analyze the load distribution between versions based on the running logs and trigger resource rebalancing. Dynamically adjust the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, switch traffic to the stable version and update the routing decision table, and record the version status change information.

[0139] As described above, the electronic device provided in this application, through an innovative version evaluation system, achieves effective status monitoring via indicator analysis and stability scoring. It constructs a traffic control mechanism, combining routing decisions and resource scheduling to establish a reliable release strategy. Adaptive adjustment is introduced, ensuring release stability through status monitoring and rule optimization. This method effectively addresses the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large-scale model services.

[0140] In another embodiment, the canary deployment and traffic control device for large model services can be configured separately from the central processing unit 9100. For example, the canary deployment and traffic control device for large model services can be configured as a chip connected to the central processing unit 9100, and the canary deployment and traffic control method function of large model services can be implemented through the control of the central processing unit.

[0141] As shown in Figure 3, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily include all the components shown in Figure 3; furthermore, the electronic device 9600 may also include components not shown in Figure 3, as can be found in existing technologies.

[0142] As shown in Figure 3, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 9100 receives input and controls the operation of various components of the electronic device 9600.

[0143] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0144] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0145] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0146] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device for communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0147] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0148] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0149] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the canary release and traffic control method for large model services with server or client execution subjects in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the canary release and traffic control method for large model services with server or client execution subjects in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0150] Step S101: Collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. Establish a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Construct a version status evaluation matrix based on the operational indicator data. Calculate the version stability score based on the version status evaluation matrix. Perform cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0151] Step S102: Based on the version stability score, assign initial traffic weights to multiple versions within the model service group, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability, generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0152] Step S103: Deploy the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Analyze the load distribution between versions based on the running logs and trigger resource rebalancing. Dynamically adjust the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, switch traffic to the stable version and update the routing decision table, and record the version status change information.

[0153] As described above, the computer-readable storage medium provided in this application embodiment achieves effective status monitoring through an innovative version evaluation system design, indicator analysis, and stability scoring. It constructs a traffic control mechanism, combining routing decisions and resource scheduling to establish a reliable release strategy. Adaptive adjustment is introduced, ensuring release stability through status monitoring and rule optimization. This method effectively addresses the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large-scale model services.

[0154] Embodiments of this application also provide a computer program product capable of implementing all steps of the canary release and traffic control method for large model services in the above embodiments, where the execution subject is a server or a client. When executed by a processor, this computer program / instruction implements the steps of the canary release and traffic control method for large model services. For example, the computer program / instruction implements the following steps:

[0155] Step S101: Collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns. Establish a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota. Construct a version status evaluation matrix based on the operational indicator data. Calculate the version stability score based on the version status evaluation matrix. Perform cluster analysis on the traffic fluctuation patterns to generate an abnormal traffic feature library.

[0156] Step S102: Based on the version stability score, assign initial traffic weights to multiple versions within the model service group, generate a routing decision table containing version identifiers, weight values, and session persistence rules, construct a computing resource scheduling strategy between versions, dynamically allocate inference nodes according to the computing resource scheduling strategy, calculate version selection probability, generate a canary release scheme containing version allocation sequence, resource configuration parameters, and adaptive adjustment rules, and use the abnormal traffic feature library to filter abnormal requests and limit traffic.

[0157] Step S103: Deploy the canary release scheme to the inference service gateway, monitor the version running status and generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Analyze the load distribution between versions based on the running logs and trigger resource rebalancing. Dynamically adjust the canary ratio according to the adaptive adjustment rules. When the rollback condition is triggered, switch traffic to the stable version and update the routing decision table, and record the version status change information.

[0158] As described above, the computer program product provided in this application, through an innovative version evaluation system, achieves effective status monitoring via indicator analysis and stability scoring. It constructs a traffic control mechanism, combining routing decisions and resource scheduling to establish a reliable release strategy. Adaptive adjustment is introduced, ensuring release stability through status monitoring and rule optimization. This method effectively addresses the shortcomings of traditional technologies in version evaluation, traffic control, and dynamic adjustment, providing technical support for large-scale model services.

[0159] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0161] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0162] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0163] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for canary deployment and traffic control of a large model service, characterized in that, The method includes: collecting operational indicator data of a model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns; establishing a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota; based on the operational indicator data, constructing a state matrix containing multi-dimensional evaluation features by calculating the indicator mean, variance, trend slope, and mutation point location; normalizing the state matrix to generate a version state evaluation matrix; calculating a version stability score based on the version state evaluation matrix; performing cluster analysis on the traffic fluctuation patterns using a density clustering algorithm to generate an abnormal traffic feature library; constructing a normalized weight vector based on the version stability score and performing smoothing; allocating initial traffic weights to multiple versions within the model service group using this vector; and generating a system containing version labels. The system uses a routing decision table based on identification, weight values, and session persistence rules to construct a computing resource scheduling strategy between versions. Inference nodes are dynamically allocated according to this strategy, with each version dynamically assigned an inference node. A node allocation mapping table is generated, and the version selection probability is calculated. A canary release scheme containing version allocation sequences, resource configuration parameters, and adaptive adjustment rules is generated. The abnormal traffic feature library is used to filter abnormal requests and limit traffic. The canary release scheme is then distributed to the inference service gateway to monitor version operation status. Operation logs are generated, including request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Based on these logs, the load distribution between versions is analyzed, and resource rebalancing is triggered. The canary release ratio is dynamically adjusted according to the adaptive adjustment rules. When a rollback condition is triggered, traffic is switched to a stable version, the routing decision table is updated, and version status change information is recorded.

2. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The collected model service group's operational metrics data includes request success rate, average inference latency, business conversion metrics, model output distribution characteristics, and traffic fluctuation patterns. A model service group with multiple versions is established, each version containing model weight file paths, inference container configuration information, and computing resource quotas. This includes: collecting real-time operational data streams from the inference service gateway, extracting request processing result identifiers, inference computation time, target task completion markers, prediction result probability distribution, and request arrival time series; segmenting and aggregating the operational data streams according to fixed time windows; calculating request success rate, average inference latency, business conversion metrics, model output distribution characteristics, and traffic fluctuation patterns; and constructing an operational metrics dataset. This operational metrics dataset is then mapped to a version management system, creating a model service group structure. Each version is assigned a globally unique identifier, which is written into the model weight file path, inference container configuration information, and computing resource quotas. A version dependency graph is established based on the globally unique identifier, generating a related data structure containing basic version information and operational metrics.

3. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The process involves: constructing a state matrix containing multi-dimensional evaluation features based on the operational indicator data by calculating the indicator mean, variance, trend slope, and mutation point location; normalizing the state matrix to generate a version state evaluation matrix; calculating a version stability score based on the version state evaluation matrix; and performing cluster analysis on the traffic fluctuation pattern using a density clustering algorithm to generate an abnormal traffic feature library. This includes: grouping operational indicator data according to version identifiers; extracting the indicator time series for each version; calculating the indicator mean, variance, trend slope, and mutation point location; constructing a state matrix containing multi-dimensional evaluation features; normalizing the state matrix to generate a version state evaluation matrix; calculating a version stability score based on the version state evaluation matrix; extracting request frequency, resource consumption, and time distribution features from the traffic fluctuation pattern; grouping the features using a density clustering algorithm; calculating the center vector and boundary threshold of each cluster; combining the center vector and boundary threshold to form an abnormal traffic feature library; and establishing a feature matching rule set.

4. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The process of constructing a normalized weight vector based on the version stability score and smoothing it to allocate initial traffic weights to multiple versions within the model service group, generating a routing decision table containing version identifiers, weight values, and session persistence rules, and constructing an inter-version computing resource scheduling strategy includes: constructing a normalized weight vector based on the version stability score; smoothing the weight vector to eliminate weight mutations; generating a version priority sequence based on the version release time; calculating the initial traffic weight allocation ratio according to the version priority sequence; establishing a mapping relationship between session identifiers and versions; converting the initial traffic weight allocation ratio into a cumulative weight interval; generating a routing decision table containing version identifiers, interval boundaries, and session persistence rules; constructing an inter-version resource scheduling strategy based on the computing node load status; and writing the resource scheduling strategy into the scheduling constraint field of the routing decision table.

5. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The process of dynamically allocating inference nodes according to the computing resource scheduling strategy, calculating version selection probabilities to generate a canary release scheme containing version allocation sequences, resource configuration parameters, and adaptive adjustment rules, and using the abnormal traffic feature library to filter abnormal requests and limit traffic includes: obtaining the computing load level of each node based on the computing resource scheduling strategy, grouping and marking inference nodes, calculating the available computing power capacity and network bandwidth of the node group, dynamically allocating inference nodes for each version according to the available computing power capacity and network bandwidth, generating a node allocation mapping table; calculating the version selection probability using the node allocation mapping table, constructing a canary release scheme containing version allocation sequences, resource configuration parameters, and adaptive adjustment rules, importing the abnormal traffic feature library into the traffic filtering module, setting request filtering thresholds and rate limiting trigger conditions based on the abnormal traffic feature library, and generating a traffic control rule set.

6. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The process of distributing the canary release plan to the inference service gateway, monitoring the version's running status to generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization, and analyzing the load distribution between versions based on the running logs and triggering resource rebalancing includes: parsing the canary release plan into gateway configuration instructions, updating the routing rule table based on the gateway configuration instructions, starting the version running status collector, collecting request volume change curves, error rate time series, performance index fluctuations, and resource usage levels, and generating running logs containing multi-dimensional monitoring data; performing time-series analysis on the running logs, calculating the differences in request distribution and resource utilization deviations between versions, constructing a load balancing score index, judging the degree of resource allocation imbalance based on the load balancing score index, triggering resource rebalancing operations when exceeding a preset threshold, and updating the node allocation plan.

7. The method for canary release and traffic control of large model services according to claim 1, characterized in that, The process involves dynamically adjusting the grayscale ratio according to the adaptive adjustment rules, switching traffic to the stable version and updating the routing decision table when a rollback condition is triggered, and recording version status change information. This includes: analyzing the current grayscale release progress based on the adaptive adjustment rules, calculating the performance index scores for each version, adjusting the version traffic allocation ratio according to the performance index scores, generating a new grayscale weight vector, and writing the grayscale weight vector into the routing decision table; monitoring version operation indicators to trigger rollback determination, pausing new version traffic allocation when a performance degradation indicator exceeds a threshold, switching all request traffic to the stable version, updating the routing decision table configuration items, and recording version status change logs including the rollback reason, switching time, and scope of impact.

8. A device for canary deployment and traffic control of a large model service, characterized in that, The device includes: a service monitoring module, used to collect operational indicator data of the model service group, including request success rate, average inference latency, business conversion indicators, model output distribution characteristics, and traffic fluctuation patterns; establishing a model service group containing multiple versions, each version including model weight file path, inference container configuration information, and computing resource quota; based on the operational indicator data, constructing a state matrix containing multi-dimensional evaluation features by calculating the indicator mean, variance, trend slope, and mutation point location; normalizing the state matrix to generate a version state evaluation matrix; calculating a version stability score based on the version state evaluation matrix; and performing cluster analysis on the traffic fluctuation patterns using a density clustering algorithm to generate an abnormal traffic feature library; and a canary release module, used to construct a normalized weight vector based on the version stability score and perform smoothing processing, thereby allocating initial traffic weights to multiple versions within the model service group and generating... A routing decision table containing version identifiers, weight values, and session persistence rules is used to construct a computing resource scheduling strategy between versions. Inference nodes are dynamically allocated according to this strategy, and a node allocation mapping table is generated for each version. The version selection probability is calculated, and a canary release scheme containing version allocation sequences, resource configuration parameters, and adaptive adjustment rules is generated. The abnormal traffic feature library is used to filter abnormal requests and limit traffic. A traffic control module is used to distribute the canary release scheme to the inference service gateway, monitor the version's running status, generate running logs containing request volume changes, error rate statistics, performance fluctuation data, and resource utilization. Based on the running logs, the load distribution between versions is analyzed and resource rebalancing is triggered. The canary ratio is dynamically adjusted according to the adaptive adjustment rules. When a rollback condition is triggered, traffic is switched to a stable version, the routing decision table is updated, and version status change information is recorded.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the canary release and traffic control method for large model services as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the canary release and traffic control method for large model services as described in any one of claims 1 to 7.

Images