Method and system for detecting and repairing output drift of large models based on historical statistics

Abstract

The application discloses a large model output drift detection and repair method and system based on historical statistics. The method comprises the following steps: collecting historical output samples at the initial stage of model deployment, and constructing a historical statistical distribution baseline on each output dimension; sampling the current output at a preset period in the running stage, and extracting feature values of each dimension; calculating a drift metric value based on the baseline, comparing the drift metric value with a dynamic threshold to determine whether a drift occurs; identifying the drift type if the drift occurs; matching and executing a repair action from a strategy library according to the drift type; feeding back the output after the repair, and updating the baseline. The application can realize lightweight and continuous output drift online detection without invading the inside of the model, and automatically execute targeted repair according to the drift type, thereby effectively improving the output consistency and long-term reliability of the large model in the production environment.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, specifically to a method and system for detecting and repairing large model output drift based on historical statistics. Background Technology

[0002] After a large language model (LLM) is deployed in a production environment, its output behavior will gradually change over time. This phenomenon is called output drift or behavioral drift. Output drift manifests as the model's output gradually deviating from the feature distribution of its initial deployment state in terms of semantics, style, content quality, or formatting, leading to a decrease in the model's reliability and consistency over time.

[0003] Output drift can occur for a variety of reasons. First, iterative updates to the model itself—such as version upgrades, parameter fine-tuning, or hotfixes—may introduce output features inconsistent with the original behavioral patterns. Second, the natural evolution of user prompts—including changes in wording, formatting adjustments, or implicit changes in task requirements—can also cause shifts in the model's output distribution. Furthermore, changes in the deployment environment, such as adjustments to temperature parameters, modifications to system prompts, or changes to context window limitations, can all trigger systematic drift in the model's output. In long-running, large-scale inference tasks, models may also experience silent degradation in inference quality due to "loss of epistemic orientation."

[0004] Currently, the detection and repair of output drift in large models mainly face the following technical challenges.

[0005] First, there is a lack of lightweight drift detection methods for production environments. Existing drift detection methods often require activation probing of the model's hidden layer activations during inference or periodic evaluation using complex evaluation datasets. These methods are computationally expensive and highly intrusive, making it difficult to achieve continuous, real-time drift monitoring without affecting the model's normal inference services.

[0006] Second, existing detection methods lack a unified and quantifiable drift judgment standard. Some solutions use statistical indicators such as KL divergence to measure changes in output distribution, but fail to establish a dynamic threshold system based on historical statistical baselines. This results in detection results that are overly sensitive to normal fluctuations or slow to react to gradual drift. Different application scenarios have significantly different tolerances for output drift, and fixed judgment thresholds are difficult to adapt to diverse deployment needs.

[0007] Third, the detection and remediation processes are disconnected. Most existing solutions only focus on drift identification, lacking automated remediation mechanisms after drift is detected. Operators typically need to intervene manually—for example, rolling back the model version, adjusting inference parameters, or redesigning prompts—this manual remediation method is slow, costly, and unable to promptly curb the continued deterioration of drift. Some studies have proposed remediation strategies such as prompt migration, but remediation decisions lack statistical support based on historical drift patterns, making it difficult to achieve accurate and adaptive remediation.

[0008] Fourth, there is a lack of refined differentiation and targeted repair of drift types. Output drift can be further subdivided into several subtypes, including semantic drift (output content deviates from the expected meaning), style drift (changes in output tone, format, or expression), and factual drift (output content is inconsistent with factual basis). Different types of drift require different repair strategies, but existing methods often adopt a "one-size-fits-all" approach, resulting in limited repair effectiveness.

[0009] In summary, how to provide a method and system that can perform lightweight and continuous detection of output drift of large models based on historical statistical distribution, and automatically perform targeted repairs based on the detection results, so as to improve the output consistency and reliability of large models in production environments, has become an urgent technical problem to be solved in this field. Summary of the Invention

[0010] This application provides a method and system for detecting and repairing large model output drift based on historical statistics, which at least addresses the problems existing in the prior art.

[0011] A first aspect of this application provides a method for detecting and repairing large model output drift based on historical statistics, including: S1: In the initial stage after model deployment, collect historical output samples of the large model under the preset prompt word set, and construct the historical statistical distribution baseline of each output dimension; S2: During the model running phase, collect the current output samples at a preset sampling period and extract the feature values ​​of the current output in each statistical dimension; S3: Based on the historical statistical distribution baseline, calculate the drift metric of the current output feature value, and determine whether output drift has occurred based on the comparison result of the drift metric and the dynamic threshold. S4: When output drift is detected, the drift type is identified in the current output to determine the category to which the drift belongs; S5: Based on the drift type, match the corresponding repair action from the preset repair strategy library and perform the repair operation; S6: Feed the repaired output back to the output buffer, write the repair record to the historical statistics database, and update the historical statistics distribution baseline.

[0012] In this embodiment, a statistical distribution baseline based on historical output samples is established in the early stage of large model deployment, providing a quantifiable reference benchmark for subsequent drift detection; continuous and lightweight monitoring of output drift is achieved using dynamic threshold technology, without intruding on the internal structure of the model or relying on external labeled data; through the linkage mechanism of drift type identification and strategy-based repair, an automated closed loop from detection to repair is realized, effectively improving the output consistency and long-term reliability of large models in the production environment.

[0013] In some embodiments of this application, the construction of historical statistical distribution baselines for each output dimension in step S1 includes: S11: Determine the output statistical dimensions that need to be monitored. The statistical dimensions include at least one or more of the following: output length, lexical diversity, punctuation usage frequency, sentence structure distribution, sentiment polarity distribution, and frequency of occurrence of keywords in a specific domain. S12: For each statistical dimension, collect the statistical values ​​of a preset number of historical output samples to form a historical statistical value sequence for that dimension; S13: Perform a normality test on the historical statistical value series for each dimension. For dimensions that conform to a normal distribution, calculate their mean and standard deviation as baseline parameters. For dimensions that do not conform to a normal distribution, use quantile regression or kernel density estimation to establish a nonparametric statistical baseline. S14: Combine and store the baseline parameters of each dimension to form the historical statistical distribution baseline.

[0014] In this embodiment, by extracting multi-dimensional statistical features from the output, the output behavior pattern of the model under normal conditions can be comprehensively characterized. Parametric or non-parametric baseline modeling methods are adopted for statistical dimensions with different distribution characteristics, which improves the adaptability of the baseline to different data distribution characteristics and provides an accurate and reliable reference for subsequent drift detection.

[0015] In some embodiments of this application, step S2, which involves extracting the feature values ​​of the current output across various statistical dimensions, includes: S21: Perform sentence segmentation and word segmentation preprocessing on the current output text to obtain a structured text unit sequence; S22: Traverse the sequence of text units and calculate the feature statistics of the current output in each statistical dimension; S23: Arrange the feature statistics of each dimension according to the same dimensions and order as the historical statistical distribution baseline to form the current output feature vector.

[0016] In some embodiments of this application, the step S3 of calculating the drift metric of the current output feature value includes: S31: For each statistical dimension, calculate the deviation between the current feature value and the baseline, based on the baseline type of that dimension; S32: Normalize the deviation values ​​to obtain standardized deviation scores for each dimension; S33: Weighted aggregation of the standardized deviation scores of each dimension to obtain the comprehensive drift metric of the current output; S34: The dynamic threshold is determined by calculating the dynamic threshold based on the dispersion of the historical statistical distribution baseline and a preset tolerance coefficient. The dynamic threshold is adaptively adjusted as the baseline is updated.

[0017] In this embodiment, by calculating the degree of deviation in each dimension and performing weighted aggregation, the degree of output drift can be comprehensively evaluated from multiple dimensions, avoiding misjudgment in a single dimension; the dynamic threshold is adaptively adjusted with the baseline update, enabling the detection system to follow the reasonable evolution of the model output distribution and reduce the false alarm rate.

[0018] In some embodiments of this application, the determination of whether output drift has occurred in step S3 is specifically as follows: When the overall drift metric exceeds the dynamic threshold, it is determined that the current output has drifted; otherwise, it is determined that the current output has not drifted. When the comprehensive drift metric value of a consecutive preset number of sampling periods does not exceed the dynamic threshold, the current output sample is included in the historical statistical database, triggering an incremental update of the historical statistical distribution baseline.

[0019] In this embodiment, the continuous exceedance judgment mechanism avoids false triggering caused by a single fluctuation; the incremental update mechanism of normal samples enables the baseline to slowly follow the reasonable evolution of the model output, avoiding the decline in detection performance caused by baseline aging.

[0020] In some embodiments of this application, the drift type identification in step S4 includes: S41: Extract the deviation direction and magnitude of the current output in each statistical dimension to form a drift feature vector; S42: Input the drift feature vector into a pre-trained drift classifier, which is trained based on historical drift samples and outputs the probability distribution of the drift belonging to each category; S43: Based on the probability distribution, the category with the highest probability is determined as the type of the current drift; The categories of drift include at least: semantic drift, style drift, factual drift, and format drift.

[0021] In this embodiment, a classifier based on drift feature vectors is used to identify drift types, which can finely distinguish different types of output drift, providing type information for subsequent targeted repair and improving the accuracy and effectiveness of the repair.

[0022] In some embodiments of this application, the preset repair strategy library mentioned in step S5 includes repair actions corresponding to each drift type, including: For semantic drift, the corresponding repair action is to adjust the model decoding temperature parameter and / or top_p sampling parameter, and regenerate the output; For style drift, the corresponding fix is ​​to insert a style constraint instruction into the system prompt and regenerate the output. For factual drift, the corresponding remedial action is to trigger an external knowledge base search, inject the search results as contextual enhancement information into the prompt words, and regenerate the output; For format drift, the corresponding repair action is to apply a format normalization filter during the post-processing stage to correct the format of the output.

[0023] In this embodiment, differentiated repair strategies are configured for different drift types, achieving precise repair based on the specific drift type. By combining various repair methods such as parameter adjustment, prompt word enhancement, external knowledge injection, and post-processing filtering, the success rate and robustness of the repair are improved.

[0024] In some embodiments of this application, after performing the repair operation in step S5, the method further includes verifying the repair effect: S51: Perform drift detection on the repaired output again and calculate the comprehensive drift metric of the repaired output; S52: If the comprehensive drift metric value of the repaired output is lower than the dynamic threshold, the repair is determined to be successful, and the repaired output is taken as the final output. S53: If the comprehensive drift metric value output after the repair is still not lower than the dynamic threshold, the repair is determined to have failed, and a backup repair strategy is triggered or a manual intervention alarm signal is generated.

[0025] In this embodiment, by introducing an effect verification mechanism after repair, a closed loop of "detection-repair-verification" is formed to ensure that the repair operation has indeed achieved the effect of eliminating drift; when the repair fails, a backup strategy or manual intervention is triggered in a timely manner to avoid the continued existence of output quality problems caused by ineffective repair.

[0026] In some embodiments of this application, updating the historical statistical distribution baseline in step S6 includes: S61: Store the output after a successful repair, along with its corresponding original prompt, repair action type, and repair timestamp, in the historical statistics database. S62: Recalculate the statistical baseline parameters for each dimension based on all historical samples in the historical statistical database according to the preset baseline update cycle; S63: Replace the original baseline parameters with the recalculated baseline parameters to complete the periodic update of the baseline.

[0027] In this embodiment of the application, by incorporating successfully repaired samples into the historical database and periodically recalculating the baseline, the baseline can continuously absorb the reasonable evolution trend of the model output, avoiding the increase in false detection rate caused by baseline solidification; at the same time, the repair records are archived as the data basis for subsequent drift pattern analysis and strategy optimization.

[0028] A second aspect of this application provides a system for detecting and repairing large model output drift based on historical statistics, comprising: The output acquisition module is used to acquire the current output samples of the large model; The statistical feature extraction module is used to extract the feature values ​​of the output sample in each statistical dimension; The baseline storage module is used to store historical statistical distribution baselines constructed based on historical output samples; The drift detection module is used to calculate the drift metric based on the historical statistical distribution baseline and the current output feature value, and to determine whether output drift has occurred. The drift type identification module is used to identify the category to which the drift belongs when output drift occurs; The repair execution module is used to match the corresponding repair action from the preset repair strategy library according to the drift type and perform the repair operation; The baseline update module is used to update the historical statistical distribution baseline based on the output after successful repair and historical samples; The processor is communicatively connected to the output acquisition module, statistical feature extraction module, baseline storage module, drift detection module, drift type identification module, repair execution module, and baseline update module, respectively, and the processor is configured to execute the method of this application.

[0029] In this embodiment, by integrating output acquisition, statistical feature extraction, baseline storage, drift detection, drift type identification, repair execution, and baseline update into a closed-loop system, the automated detection and repair of large model output drift is realized. After the processor executes the method of this application, it can achieve lightweight and continuous drift monitoring without intruding on the internal structure of the model, and automatically perform targeted repairs according to the drift type, which significantly improves the output consistency and long-term reliability of large models in the production environment. Attached Figure Description

[0030] Figure 1A flowchart illustrating a method for detecting and repairing large model output drift based on historical statistics, provided in an embodiment of this application; Figure 2 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0032] To make the purpose, technical solution, and advantages of this application clearer, the following will be described in conjunction with the appendix. Figure 1-2 The following is an explanation using specific examples.

[0033] Continuous monitoring of output quality is crucial for ensuring application reliability during the production and deployment of large models. However, through long-term engineering practice, the inventors have discovered that existing large model output monitoring technologies face a series of deep-seated contradictions in practical applications. First, the output behavior of large models is non-deterministic; the same prompt word may produce different outputs at different times and under different contexts, making traditional quality detection methods based on fixed thresholds difficult to adapt to the inherent fluctuations in the output itself. Second, output drift is often gradual—the model may gradually deviate from its initial behavior pattern over days or weeks; this slow, gradual change, like "boiling a frog in lukewarm water," is difficult to detect through manual inspection or simple rule detection. Even more challenging is that different types of drift (semantic drift, style drift, factual drift, etc.) have different causes and characteristics, requiring differentiated repair strategies, while existing solutions lack the ability to handle such heterogeneous drift with fine precision. Furthermore, even when drift is detected, existing systems typically only issue alarms and cannot automatically repair the problem; operators need to manually investigate the cause and adjust the model configuration, resulting in slow response times and high costs.

[0034] Through in-depth analysis, the inventors discovered that the root cause of the above problems lies in three technological gaps in existing technologies. First, there is a lack of a unified, quantifiable system for representing output behavior. The output of large models is high-dimensional, unstructured natural language, making it difficult to directly measure drift mathematically. A mapping bridge from unstructured text to structured statistical features is needed. Second, there is a lack of dynamic reference benchmarks based on historical data. Output drift is essentially a deviation of the current output distribution from the historical normal distribution. However, existing solutions mostly use static thresholds or external reference standards, failing to fully utilize the continuously accumulated historical output data after deployment to build an adaptive baseline. Third, the detection and repair processes are disconnected. The detection process only outputs a binary judgment of "whether there is drift," lacking refined identification of drift types. This prevents the repair process from taking targeted measures, often resorting to general retry or rollback strategies with limited repair effectiveness.

[0035] To address the aforementioned technological gap, the inventors proposed a progressive technical solution. First, they introduced multi-dimensional statistical feature extraction techniques to transform the unstructured text output by the large model into quantifiable statistical feature vectors, including multiple dimensions such as output length, lexical diversity, sentence structure, and sentiment polarity, thus laying the mathematical foundation for drift measurement. Second, recognizing that the historical output of the large model after deployment contains statistical information about its normal behavioral patterns, they designed a drift detection framework based on a historical statistical distribution baseline: In the initial deployment phase, a sufficient number of historical output samples are collected to construct statistical baselines for each dimension; during the runtime phase, the current output features are compared with the baselines, and the degree of drift is quantified by calculating a weighted aggregate value of the multi-dimensional deviation; adaptive judgment is achieved using dynamic thresholding technology. Building upon this, the inventors further introduced a classifier based on the drift feature vectors to perform refined type identification of the detected drift—distinguishing between semantic drift, style drift, factual drift, and format drift—and pre-configured targeted repair strategies for each type (including parameter adjustment, cue word enhancement, external knowledge injection, and post-processing filtering). Finally, to address the baseline aging issue and create a closed loop for continuous improvement, the inventors incorporated the output after successful repair into the historical database. By periodically recalculating the baseline parameters, they achieved incremental updates to the baseline. Simultaneously, they archived the repair records as the data foundation for subsequent drift pattern analysis and strategy optimization, thus forming a complete technical closed loop from multidimensional feature extraction, historical baseline construction, dynamic drift detection, refined type recognition to strategy-based repair and continuous baseline evolution.

[0036] Please refer to Figure 1 , Figure 1 As a first aspect of the embodiments of this application, a method for detecting and repairing large model output drift based on historical statistics is provided, including: S1: In the initial stage after model deployment, collect historical output samples of the large model under the preset prompt word set, and construct the historical statistical distribution baseline of each output dimension; S2: During the model running phase, collect the current output samples at a preset sampling period and extract the feature values ​​of the current output in each statistical dimension; S3: Based on the historical statistical distribution baseline, calculate the drift metric of the current output feature value, and determine whether output drift has occurred based on the comparison result of the drift metric and the dynamic threshold. S4: When output drift is detected, the drift type is identified in the current output to determine the category to which the drift belongs; S5: Based on the drift type, match the corresponding repair action from the preset repair strategy library and perform the repair operation; S6: Feed the repaired output back to the output buffer, write the repair record to the historical statistics database, and update the historical statistics distribution baseline.

[0037] In this embodiment, step S1, the initial stage after model deployment refers to a period of time after the large model is first launched and run. During this stage, the model's behavior has not yet experienced significant drift, and its output can represent the model's behavior pattern under normal conditions. The preset prompt word set refers to a pre-designed set of prompt word samples covering various typical tasks in the model's expected application scenarios. This set should have sufficient diversity and representativeness to ensure that the collected historical output samples can comprehensively reflect the normal output characteristics of the model under various task types. Historical output samples refer to the corresponding output text generated by the model after inputting each prompt word from the preset prompt word set into the large model. The historical statistical distribution baseline refers to the reference distribution parameters obtained after statistical analysis of the historical output samples across various statistical dimensions. This baseline serves as the comparison benchmark for subsequent drift detection.

[0038] In step S2, the preset sampling period refers to a pre-defined time interval for periodically sampling the model output. This period can be configured according to the real-time requirements of the application scenario; for example, it can be set to sample once every 100 inference requests or once every 5 minutes. The current output sample refers to the output text obtained from the model at the current sampling moment. Extracting the feature values ​​of the current output in each statistical dimension means calculating the features of the current output text according to the same statistical dimension system as in step S1 to obtain a set of quantifiable feature values.

[0039] In step S3, the drift metric is a comprehensive index value used to quantify the degree of deviation of the current output from the historical baseline. The larger the value, the more significant the difference between the current output and the historical normal pattern. The dynamic threshold is a critical value used to determine whether drift has occurred. This threshold is not fixed, but is dynamically calculated based on the dispersion of the historical statistical distribution baseline and a preset tolerance coefficient, so that the detection standard can adaptively follow the changes in the baseline.

[0040] In step S4, drift type identification refers to further analyzing the nature and category of the drift after confirming its occurrence. Different types of drift exhibit different deviation patterns in statistical characteristics—for example, semantic drift may manifest as a systematic shift in keyword distribution and sentiment polarity, while style drift may manifest as changes in sentence structure and lexical diversity—by analyzing the patterns of deviation direction and magnitude, the category to which the drift belongs can be inferred.

[0041] In step S5, the repair strategy library refers to a pre-built knowledge base that associates each drift type with its corresponding repair action. A repair action is a specific operation performed to eliminate or suppress drift, including but not limited to adjusting model inference parameters, modifying system prompts, injecting external knowledge, or applying post-processing filters.

[0042] In step S6, feeding the repaired output back to the output cache means returning the output text processed by the repair operation as the final result to the caller or downstream system. The repair record refers to structured data containing information such as the original prompt word, original output, repaired output, drift type, repair action type, and timestamp. Updating the historical statistical distribution baseline means incorporating the successfully repaired output samples into the historical statistical database and recalculating the statistical baseline parameters for each dimension based on the updated database, ensuring that the baseline continuously reflects the model's normal behavior pattern in the current state.

[0043] In this application, firstly, step S1 collects historical output samples and constructs a statistical distribution baseline in the initial stage after model deployment. This step establishes a reference benchmark for the entire detection system. The output of a large model has inherent randomness and volatility. Without a statistical baseline based on historical data as a reference, any judgment of anomalies in a single output lacks objective basis. By collecting a sufficient number of historical samples and establishing a multi-dimensional statistical distribution, step S1 quantifies the model's behavior pattern under normal conditions into calculable mathematical parameters, providing a reliable comparison object for subsequent drift measurement.

[0044] Step S2 involves collecting the current output and extracting feature values ​​during the model runtime phase at a preset sampling period. This step transforms unstructured text into structured feature vectors. Since the output of a large model is natural language text, direct mathematical drift calculations are not feasible. By extracting multi-dimensional statistical features such as output length, lexical diversity, and sentence structure, step S2 maps the text to a quantifiable and computable feature space, making drift detection mathematically operable. The introduction of the sampling period allows detection to be performed continuously and periodically, enabling continuous monitoring of the model's output behavior.

[0045] Step S3 calculates the drift metric of the current output based on the historical statistical distribution baseline and compares it with the dynamic threshold, completing the leap from feature difference to drift determination. By calculating the weighted aggregate value of the deviation degree of each dimension, step S3 compresses the multidimensional feature difference into a comprehensive drift metric, making the degree of drift intuitively quantifiable and comparable. The introduction of the dynamic threshold allows the determination criterion to adaptively adjust with changes in the baseline—when the baseline changes slowly due to the reasonable evolution of the model output, the threshold also adjusts accordingly, avoiding the "crying wolf" effect caused by a fixed threshold.

[0046] Step S4 identifies the type of drift after determining that a drift has occurred, advancing drift detection from a binary judgment of "whether a drift has occurred" to a more refined diagnosis of "what kind of drift has occurred." Different types of drift have different causes and characteristics. By analyzing the direction and magnitude patterns of deviations in various dimensions, step S4 can distinguish between different categories such as semantic drift, style drift, factual drift, and format drift, providing type information for subsequent targeted remediation.

[0047] Step S5 matches and executes the corresponding repair action based on the drift type, achieving automated connection from diagnosis to treatment. Adjusting decoding parameters for semantic drift, injecting style constraints for style drift, triggering knowledge base retrieval for factual drift, and applying post-processing filters for format drift—each repair strategy is optimized for the causes and characteristics of a specific drift type, resulting in a higher repair success rate compared to general retry or rollback strategies.

[0048] Step S6 feeds the repaired output back to the output cache and writes the repair record to the historical database, completing the final step of the closed loop. By incorporating the successfully repaired output into the historical statistical database and periodically recalculating the baseline, step S6 ensures that the baseline can continuously absorb the reasonable evolution of the model output, avoiding the performance degradation caused by baseline aging. Simultaneously, archiving the repair record provides a data foundation for subsequent drift pattern analysis and strategy optimization. These six steps—baseline construction, periodic sampling, drift measurement and determination, type identification, strategy-based repair, and baseline update—form a complete "monitoring-diagnosis-treatment-recovery" closed loop. Each step is interconnected and mutually supportive through data transfer and feedback, achieving automated detection and repair of large model output drift.

[0049] In some embodiments of this application, the construction of historical statistical distribution baselines for each output dimension in step S1 includes: S11: Determine the output statistical dimensions that need to be monitored. The statistical dimensions include at least one or more of the following: output length, lexical diversity, punctuation usage frequency, sentence structure distribution, sentiment polarity distribution, and frequency of occurrence of keywords in a specific domain. S12: For each statistical dimension, collect the statistical values ​​of a preset number of historical output samples to form a historical statistical value sequence for that dimension; S13: Perform a normality test on the historical statistical value series for each dimension. For dimensions that conform to a normal distribution, calculate their mean and standard deviation as baseline parameters. For dimensions that do not conform to a normal distribution, use quantile regression or kernel density estimation to establish a nonparametric statistical baseline. S14: Combine and store the baseline parameters of each dimension to form the historical statistical distribution baseline.

[0050] In this embodiment, in step S11, output length refers to the number of characters, words, or tokens in the output text, reflecting the level of detail in the model's output. Lexical diversity refers to the ratio of the number of different words in the output text to the total number of words, reflecting the richness of the model's vocabulary; it can be measured using the Type-Token Ratio (TR) or its variants. Punctuation frequency refers to the number or proportion of various punctuation marks appearing in the output text, reflecting the model's expression habits and sentence rhythm. Sentence structure distribution refers to the proportion of different sentence types (such as declarative, interrogative, imperative, and exclamatory sentences) in the output text, reflecting the model's expression style. Sentiment polarity distribution refers to the distribution of sentiment in the output text, which can be obtained by using a sentiment analysis model to score the output text for sentiment. Domain-specific keyword frequency refers to the number or density of domain terms or keywords related to the application scenario appearing in the output, reflecting the degree to which the model utilizes domain knowledge. The selection of the above statistical dimensions is based on consideration of the sensitivity to different types of drift: semantic drift is often accompanied by changes in keyword distribution and sentiment polarity, style drift is usually reflected in changes in sentence structure and vocabulary diversity, and factual drift may manifest as abnormal fluctuations in the frequency of domain keywords.

[0051] In step S12, the preset quantity refers to the minimum number of historical samples that need to be collected to ensure statistical significance. This quantity can be determined based on statistical power analysis, and it is generally recommended to have no less than 30 samples, preferably 100 to 500 samples. The historical statistical value sequence refers to a one-dimensional array formed by arranging the statistical values ​​of each historical output sample in the same dimension in chronological order.

[0052] In step S13, the normality test refers to the statistical test method used to determine whether a data sequence follows a normal distribution. For example, the Shapiro-Wilk test, Kolmogorov-Smirnov test, or Anderson-Darling test can be used. For dimensions that conform to a normal distribution, the mean and standard deviation are used as baseline parameters—the mean is an estimate of the central location of the normal level of that dimension, and the standard deviation is an estimate of the dispersion of the normal fluctuations of that dimension. For dimensions that do not conform to a normal distribution, quantile regression or kernel density estimation is used to establish a nonparametric statistical baseline—quantile regression describes the distribution characteristics by estimating specific quantiles (such as the median or quartiles), while kernel density estimation describes the overall distribution shape through a smoothed probability density function. This classification is based on the fact that different statistical dimensions have different data generation mechanisms. Some dimensions (such as output length) may approximately follow a normal distribution, while others (such as the frequency of specific keyword occurrences) may exhibit a skewed or multimodal distribution. Using a uniform parameterization method would lead to inaccurate baseline descriptions.

[0053] In step S14, storing the baseline parameters of each dimension in combination means organizing the baseline parameters of each dimension into structured data records in a unified format and order so that they can be quickly read and retrieved in subsequent steps.

[0054] In this application, steps S11 to S14 sequentially complete the determination of monitoring dimensions, the collection of historical data, the estimation of baseline parameters, and the storage of baseline data. Step S11 determines the statistical dimension system to be monitored. The design of this system directly affects the coverage and sensitivity of subsequent drift detection—too few dimensions may miss certain types of drift, while too many dimensions may introduce noise. Step S12 collects the historical statistical value sequences of each dimension, providing a data foundation for baseline estimation. Step S13 classifies dimensions into parametric and non-parametric categories through normality tests and models them separately. This classification method makes the baseline more adaptable to different data distribution characteristics—for normally distributed dimensions, the mean and standard deviation provide a concise and effective description; for non-normally distributed dimensions, quantile regression or kernel density estimation provides a more flexible distribution description capability. Step S14 combines and stores the baseline parameters of each dimension, forming structured baseline data, providing directly callable reference parameters for drift measurement calculation in step S3. The above four steps, from dimension design, data collection, parameter estimation to data storage, are sequentially connected, constructing a complete process for establishing a historical statistical distribution baseline.

[0055] In some embodiments of this application, step S2, which involves extracting the feature values ​​of the current output across various statistical dimensions, includes: S21: Perform sentence segmentation and word segmentation preprocessing on the current output text to obtain a structured text unit sequence; S22: Traverse the sequence of text units and calculate the feature statistics of the current output in each statistical dimension; S23: Arrange the feature statistics of each dimension according to the same dimensions and order as the historical statistical distribution baseline to form the current output feature vector.

[0056] In this embodiment, step S21, sentence preprocessing refers to segmenting the output text into independent sentence units according to sentence terminators such as periods, question marks, and exclamation marks. Word preprocessing refers to segmenting each sentence unit into independent tokens or word units according to the lexical boundaries of the language. For Chinese text, word segmentation can employ dictionary-based maximum matching or statistical sequence labeling; for languages ​​separated by spaces, such as English, word segmentation can be achieved through space segmentation combined with punctuation cleaning. Obtaining a structured text unit sequence means that the output text has been converted into a structured representation of sentence and word sequences, facilitating subsequent statistical feature calculations.

[0057] In step S22, traversing the text unit sequence refers to sequentially accessing each sentence unit and vocabulary unit processed in step S21, and calculating the corresponding statistics for each statistical dimension. For example, for the output length dimension, the total number of words or characters is counted; for the vocabulary diversity dimension, the number of different words is counted and the ratio to the total number of words is calculated; for the punctuation frequency dimension, the number of occurrences of each type of punctuation mark is counted; and for the sentence structure distribution dimension, the sentence type of each sentence is determined and the proportion of each type of sentence is counted.

[0058] In step S23, arranging according to the same dimensions and order as the historical statistical distribution baseline means organizing the feature statistics of each dimension calculated in step S22 into a vector according to the storage order of the baseline parameters in step S14, so as to ensure that the current output feature vector is strictly consistent with the historical statistical distribution baseline in terms of dimension order, so as to perform dimension-by-dimensional comparison calculations in subsequent steps.

[0059] In this application, steps S21 to S23 transform the unstructured natural language output text into structured numerical feature vectors. Sentence segmentation and word segmentation preprocessing in step S21 form the basis for all subsequent statistical calculations; the quality of the text unit sequence construction directly affects the accuracy of the calculation of statistics for each dimension. Step S22 calculates the feature statistics for each dimension on each text unit, transforming the text content into quantifiable values. Step S23 arranges the statistics for each dimension into feature vectors in a unified order, enabling a dimension-by-dimensional mathematical comparison between the current output and the historical baseline. These three steps, from text structuring and feature calculation to vector organization, are performed sequentially, ensuring that the data foundation for subsequent drift metric calculations is accurate, complete, and consistent in format.

[0060] In some embodiments of this application, the step S3 of calculating the drift metric of the current output feature value includes: S31: For each statistical dimension, calculate the deviation between the current feature value and the baseline, based on the baseline type of that dimension; S32: Normalize the deviation values ​​to obtain standardized deviation scores for each dimension; S33: Weighted aggregation of the standardized deviation scores of each dimension to obtain the comprehensive drift metric of the current output; S34: The dynamic threshold is determined by calculating the dynamic threshold based on the dispersion of the historical statistical distribution baseline and a preset tolerance coefficient. The dynamic threshold is adaptively adjusted as the baseline is updated.

[0061] In this embodiment, in step S31, the baseline type refers to whether the baseline for this dimension determined in step S13 is a parametric baseline (normal distribution) or a non-parametric baseline (non-normal distribution). For a parametric baseline, the deviation value can be calculated as the absolute value of the difference between the current feature value and the mean divided by the standard deviation, i.e., the absolute value of the standard score (z-score), which indicates how many standard deviation units the current feature value deviates from the mean. For a non-parametric baseline, the deviation value can be calculated as the quantile position of the current feature value relative to the baseline distribution, for example, the number of times the current feature value exceeds the difference between the upper and lower quartiles (IQR), or the deviation of the cumulative probability of the current feature value in the baseline distribution from 0.5 can be calculated through kernel density estimation.

[0062] In step S32, normalization refers to mapping the deviation values ​​of each dimension to a uniform numerical range to eliminate comparability issues caused by differences in units and scales between different dimensions. For example, the sigmoid function can be used to map the deviation values ​​to a range of 0 to 1, or min-max normalization can be used to map the deviation values ​​of each dimension to a uniform interval. The larger the standardized deviation score after normalization, the more severe the deviation of the current feature value of that dimension from the baseline.

[0063] In step S33, weighted aggregation refers to the weighted summation or weighted average of the standardized deviation scores of each dimension according to preset weights. The weights of each dimension can be set according to the importance of that dimension to drift detection—for example, assigning higher weights to dimensions that are more sensitive in specific application scenarios. The weights can be determined based on domain expert knowledge or by learning from the discriminative power of each dimension in historical drift samples. The comprehensive drift metric is a comprehensive reflection of the degree of deviation of each dimension; the larger the value, the more severe the overall deviation of the current output from the historical normal pattern.

[0064] In step S34, dispersion refers to statistical measures reflecting the degree of data dispersion, such as the variance, standard deviation, or interquartile range of the historical statistical distribution baseline. The tolerance coefficient is a pre-set adjustment parameter used to control detection sensitivity. A larger tolerance coefficient results in a higher dynamic threshold and less sensitive detection (increased risk of missed detection but reduced risk of false alarms); a smaller tolerance coefficient results in a lower dynamic threshold and more sensitive detection (reduced risk of missed detection but increased risk of false alarms). The adaptive adjustment of the dynamic threshold with baseline updates means that when the historical statistical distribution baseline changes due to the update in step S6, the dynamic threshold is recalculated accordingly, ensuring that the detection standard always matches the dispersion of the current baseline.

[0065] In this application, steps S31 to S34 transform multidimensional feature differences into determinable drift metrics. Step S31 employs differentiated deviation calculation methods based on the baseline type of each dimension—using standard scores for normally distributed dimensions and quantiles or kernel density for non-normally distributed dimensions—ensuring that the deviation calculation is statistically reasonable for each dimension. Step S32 eliminates dimensional differences between dimensions through normalization, allowing deviations from different dimensions to be compared and aggregated on the same scale. Step S33 compresses the deviation information from each dimension into a comprehensive index through weighted aggregation, achieving single-value quantification of drift and facilitating subsequent threshold comparisons. The dynamic threshold determined in step S34 is calculated based on the dispersion and tolerance coefficient of the baseline, ensuring that the detection standard considers both the magnitude of normal fluctuations and the trade-off between false positives and false negatives. These four steps, from dimensional deviation calculation, normalization, weighted aggregation to dynamic threshold determination, construct a complete drift measurement and determination system.

[0066] In some embodiments of this application, the determination of whether output drift has occurred in step S3 is specifically as follows: When the overall drift metric exceeds the dynamic threshold, it is determined that the current output has drifted; otherwise, it is determined that the current output has not drifted. When the comprehensive drift metric value of a consecutive preset number of sampling periods does not exceed the dynamic threshold, the current output sample is included in the historical statistical database, triggering an incremental update of the historical statistical distribution baseline.

[0067] In this embodiment, the preset number of consecutive sampling periods refers to the number of consecutive samplings required to confirm that the model output has stabilized within the normal range. It is understood that a single sampling not exceeding the dynamic threshold may be the result of random fluctuations and is insufficient to indicate that the model output has recovered to a stable state. By requiring that multiple consecutive sampling periods not exceed the threshold, the reliability of baseline updates can be improved, avoiding unnecessary baseline updates triggered by a single, accidental normal fluctuation. The preset number can be set according to the application scenario's requirements for response speed and stability; for example, it can be 3, 5, or 10 times. Incremental updates refer to correcting the baseline parameters based on the existing baseline using newly added normal samples, rather than completely recalculating. Incremental updates have the advantages of lower computational overhead and faster response speed compared to full recalculation, making them suitable for online scenarios requiring frequent baseline updates.

[0068] In this application, a continuous exceedance determination mechanism avoids drift alarms triggered by accidental fluctuations in a single output, thus improving the robustness of detection. When no drift is determined and multiple consecutive cycles are normal, the current output is added to the historical database and baseline incremental updates are triggered. This allows the baseline to continuously absorb reasonable evolutions in the model output, preventing it from gradually deviating from its current normal distribution due to prolonged lack of updates. This dual-mode working mechanism of "updating during normal periods and alarming during abnormal periods" enables the detection system to remain sensitive to real-world drift while adaptively following reasonable changes in the model output.

[0069] In some embodiments of this application, the drift type identification in step S4 includes: S41: Extract the deviation direction and magnitude of the current output in each statistical dimension to form a drift feature vector; S42: Input the drift feature vector into a pre-trained drift classifier, which is trained based on historical drift samples and outputs the probability distribution of the drift belonging to each category; S43: Based on the probability distribution, the category with the highest probability is determined as the type of the current drift; The categories of drift include at least: semantic drift, style drift, factual drift, and format drift.

[0070] In this embodiment, in step S41, the deviation direction refers to whether the current feature value is larger or smaller than the baseline center position, which can be represented by a positive or negative sign. The deviation magnitude refers to the degree to which the current feature value deviates from the baseline center position, which can be represented by the standardized deviation score calculated in step S32. The deviation direction and deviation magnitude of each dimension are combined into a vector, namely the drift feature vector, which describes the "shape" of the current drift in the multidimensional statistical space.

[0071] In step S42, the drift classifier refers to a supervised learning model whose input is the drift feature vector and whose output is the probability distribution of the drift belonging to each category. This classifier is trained based on historical drift samples—historical drift samples refer to historical drift events whose drift types have been labeled by experts. Each sample contains the drift feature vector corresponding to that drift event and its true drift type label. The specific implementation of the classifier can employ multi-class logistic regression, support vector machines, random forests, or lightweight neural networks, etc. The training process of the classifier involves optimizing the model parameters on historical drift samples to make the model's prediction of the drift type as close as possible to the true label.

[0072] In step S43, determining the category with the highest probability as the type of the current drift means making a decision using the maximum a posteriori probability criterion and selecting the category with the highest probability value in the classifier's output probability distribution as the final type determination result.

[0073] Among the categories of drift, semantic drift refers to a systematic shift in the meaning of the model's output, such as a change in the core information of the answer in a question-answering task, or a deviation from the topic in a text generation task. Style drift refers to a change in the way the model's output is expressed, such as a change in tone from formal to casual, a change in sentence structure from concise to lengthy, or a change in wording style. Factual drift refers to an inconsistency between the model's output and the factual basis, such as providing incorrect factual information in knowledge-based question answering. Format drift refers to a change in the structural format of the model's output, such as outputting in a format other than the expected JSON format, omitting necessary fields or tags, or anomalies in Markdown formatting.

[0074] In this application, steps S41 to S43 advance drift detection from a binary judgment of "whether drifting has occurred" to a refined diagnosis of "what kind of drift." Step S41 extracts the deviation direction and magnitude to form a drift feature vector. This feature vector contains not only information about the intensity of the drift but also information about the directional pattern of the drift in multidimensional space—and the directional pattern is the key basis for distinguishing different types of drift. For example, semantic drift may manifest as a unidirectional shift in keyword distribution and sentiment polarity, style drift may manifest as a coordinated change in sentence structure and lexical diversity, and factual drift may only manifest as an abnormality in the frequency of domain keywords while other dimensions remain normal. Step S42 classifies the drift feature vector using a pre-trained drift classifier, transforming the identification of directional patterns into a computable classification problem. The classifier is trained based on historical drift samples, enabling it to learn typical patterns of various types of drift in the statistical feature space. Step S43 makes a type determination based on the probability distribution output by the classifier. The above three steps, from feature extraction and classification prediction to type determination, are performed sequentially, providing accurate type information for targeted repair in the subsequent step S5.

[0075] In some embodiments of this application, the preset repair strategy library mentioned in step S5 includes repair actions corresponding to each drift type, including: For semantic drift, the corresponding repair action is to adjust the model decoding temperature parameter and / or top_p sampling parameter, and regenerate the output; For style drift, the corresponding fix is ​​to insert a style constraint instruction into the system prompt and regenerate the output. For factual drift, the corresponding remedial action is to trigger an external knowledge base search, inject the search results as contextual enhancement information into the prompt words, and regenerate the output; For format drift, the corresponding repair action is to apply a format normalization filter during the post-processing stage to correct the format of the output.

[0076] In this embodiment, for semantic drift, adjusting the model decoding temperature parameter and / or the top_p sampling parameter can change the sharpness or truncation of the model's output probability distribution, thereby affecting the randomness and diversity of the model during decoding. When semantic drift is detected, appropriately reducing the temperature parameter can make the model output more concentrated on the highest probability words, reducing semantic shifts caused by excessive randomness; or adjusting the top_p parameter can control the size of the sampling candidate set, making the output more stable and predictable. After adjusting the parameters, the model is re-invoked using the same prompt words to generate the output again to obtain the corrected result.

[0077] To address style drift, inserting style constraint instructions into the system prompts involves adding explicit language style requirements to the model's system prompts, such as "Please answer in a formal and objective tone," "Please keep your answer concise and clear," and "Please use technical terms." The insertion of style constraint instructions provides explicit behavioral guidance to the model during subsequent generation, thus pulling the output style back onto the expected track. After inserting the constraint instructions, the model is called back to generate the output.

[0078] For factual drift, triggering external knowledge base retrieval refers to initiating a search request to external knowledge bases (such as enterprise knowledge bases, Wikipedia, structured databases, etc.) when a potential factual bias is detected in the model output. This retrieves factual information relevant to the current question by using the current question or prompt words as the query statement. Injecting the search results as contextual enhancement information into the prompt words involves piecing together relevant knowledge fragments from the search results with the original prompt words. This provides the model with additional factual evidence, allowing it to regenerate its output based on external knowledge, thereby improving the factual accuracy of the output.

[0079] To address format drift, applying format normalization filters in the post-processing stage involves automatically correcting the format of the generated output text. For example, if the expected output is JSON but the model output is missing necessary parentheses or quotation marks, the format normalization filter can automatically complete the missing formatting symbols; if the expected output is a Markdown table but the formatting is incorrect, the filter can normalize the table. Format normalization filters can be implemented using rule-based methods (such as regular expression matching and replacement) or template-based methods.

[0080] In this application, differentiated repair strategies are configured for different types of drift, achieving precise repair tailored to the specific type of drift. The root cause of semantic drift lies in the excessive randomness during the decoding process, leading to outputs deviating from the expected semantic trajectory. Therefore, by adjusting decoding parameters to tighten the probability distribution of the output, the semantic direction is controlled from the generation source. Style drift is essentially due to the model lacking clear style guidance. Therefore, explicit style constraint instructions are inserted into the system prompts to provide the model with clear behavioral boundaries. Factual drift stems from insufficient or outdated knowledge reserves within the model itself. Therefore, additional factual information is introduced by triggering external knowledge base retrieval to compensate for the limitations of the model's knowledge. Format drift is a structural-level deviation that can be corrected through post-processing filters without regeneration, avoiding unnecessary inference overhead. The above four repair strategies are optimized for the causes and characteristics of different drift types, exhibiting higher repair efficiency and success rates compared to general retry strategies.

[0081] In some embodiments of this application, after performing the repair operation in step S5, the method further includes verifying the repair effect: S51: Perform drift detection on the repaired output again and calculate the comprehensive drift metric of the repaired output; S52: If the comprehensive drift metric value of the repaired output is lower than the dynamic threshold, the repair is determined to be successful, and the repaired output is taken as the final output. S53: If the comprehensive drift metric value output after the repair is still not lower than the dynamic threshold, the repair is determined to have failed, and a backup repair strategy is triggered or a manual intervention alarm signal is generated.

[0082] In this embodiment, step S51, performing drift detection on the repaired output again, means re-executing steps S2 and S3 on the output text after the repair operation, i.e., re-extracting statistical feature values ​​and calculating the comprehensive drift metric. The purpose of this step is to objectively evaluate whether the repair operation has indeed eliminated the drift.

[0083] In step S52, if the overall drift metric of the output after repair is lower than the dynamic threshold, it indicates that the repair operation has effectively pulled the output back to the normal distribution range, and the repair is successful. The repaired output is then returned as the final result. In step S53, if the overall drift metric of the output after repair is still not lower than the dynamic threshold, it indicates that the repair operation has failed to effectively eliminate the drift. In this case, a backup repair strategy is triggered—for example, trying another type of repair action (such as switching from parameter adjustment to prompt enhancement), or generating a manual intervention alarm signal to notify the operator to handle it manually.

[0084] In this application, a closed loop of "detection-repair-verification" is formed by introducing an effect verification mechanism after repair. Step S51 re-detects drift in the repaired output, using objective statistical metrics to evaluate the repair effect, avoiding reliance on subjective judgment or simple assumptions that the repair will definitely succeed. Step S52 uses the repaired output as the final output when the repair is successful, ensuring output quality. Step S53 promptly triggers backup strategies or manual intervention when the repair fails, preventing the persistence of output quality problems caused by ineffective repair. This verification mechanism makes the entire repair process observable and measurable in its effect evaluation, providing a guarantee for the reliability of the system.

[0085] In some embodiments of this application, updating the historical statistical distribution baseline in step S6 includes: S61: Store the output after a successful repair, along with its corresponding original prompt, repair action type, and repair timestamp, in the historical statistics database. S62: Recalculate the statistical baseline parameters for each dimension based on all historical samples in the historical statistical database according to the preset baseline update cycle; S63: Replace the original baseline parameters with the recalculated baseline parameters to complete the periodic update of the baseline.

[0086] In this embodiment, in step S61, the output after successful repair refers to the output text verified and confirmed as successfully repaired in step S52. The original prompt word refers to the prompt word input into the model when generating this output. The repair action type refers to the category identifier of the actual repair operation performed in step S5, such as "temperature adjustment," "prompt word enhancement," "knowledge base retrieval," or "format filtering." The repair timestamp refers to the time when the repair operation was completed. This information is associated and stored in a historical statistical database, so that each historical sample not only contains the output text and its statistical features but also its repair history, providing rich metadata for subsequent drift pattern analysis and strategy optimization.

[0087] In step S62, the preset baseline update cycle refers to a pre-defined time interval for recalculating the baseline. Since the number of samples in the historical statistical database continues to grow over time, completely recalculating the baseline every time a new sample is added can incur significant computational overhead. By setting an update cycle (e.g., updating every 1000 samples or every 24 hours), a balance can be struck between computational overhead and baseline timeliness. Recalculating the statistical baseline parameters for each dimension based on all historical samples in the historical statistical database means repeating steps S12 to S13, but using the entire dataset of all samples in the current historical statistical database.

[0088] In step S63, the recalculated baseline parameters replace the original baseline parameters, completing the periodic update of the baseline so that the baseline is always described based on all currently available historical data.

[0089] In this application, steps S61 to S63 incorporate successfully repaired samples into the historical database and achieve continuous baseline evolution through periodic recalculation. Step S61 stores successfully repaired samples and their metadata in the historical database, providing a new data source for baseline updates, while archiving repair records provides a data foundation for subsequent analysis. Step S62 recalculates baseline parameters based on all historical samples according to a preset period, enabling the baseline to continuously absorb the reasonable evolutionary trend of the model output and avoiding the decline in detection performance caused by baseline solidification. Step S63 completes the baseline replacement and update. The above three steps, from data archiving and periodic recalculation to baseline replacement, are performed sequentially, constructing a complete baseline continuous evolution mechanism, enabling the detection system to maintain an accurate description of the model output distribution during long-term operation.

[0090] A second aspect of this application provides a system for detecting and repairing large model output drift based on historical statistics, comprising: The output acquisition module is used to acquire the current output samples of the large model; The statistical feature extraction module is used to extract the feature values ​​of the output sample in each statistical dimension; The baseline storage module is used to store historical statistical distribution baselines constructed based on historical output samples; The drift detection module is used to calculate the drift metric based on the historical statistical distribution baseline and the current output feature value, and to determine whether output drift has occurred. The drift type identification module is used to identify the category to which the drift belongs when output drift occurs; The repair execution module is used to match the corresponding repair action from the preset repair strategy library according to the drift type and perform the repair operation; The baseline update module is used to update the historical statistical distribution baseline based on the output after successful repair and historical samples; The processor is communicatively connected to the output acquisition module, statistical feature extraction module, baseline storage module, drift detection module, drift type identification module, repair execution module, and baseline update module, respectively, and the processor is configured to execute the method of this application.

[0091] In this embodiment, by integrating output acquisition, statistical feature extraction, baseline storage, drift detection, drift type identification, repair execution, and baseline update into a closed-loop system, the automated detection and repair of large model output drift is realized. After the processor executes the method of this application, it can achieve lightweight and continuous drift monitoring without intruding on the internal structure of the model, and automatically perform targeted repairs according to the drift type. This enables the system to have a complete closed-loop capability from monitoring, diagnosis, repair to baseline evolution, which significantly improves the output consistency and long-term reliability of large models in the production environment.

[0092] See Figure 2 , Figure 2 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 2 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to execute the aforementioned large model output drift detection and repair method based on historical statistics.

[0093] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0094] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0095] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0096] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in any embodiment of the large model output drift detection and repair method based on historical statistics provided in the embodiments of this application, or they can execute the implementation methods of the electronic devices described in the embodiments of this application, which will not be repeated here.

[0097] In another embodiment of this application, an electronic device is provided. The electronic device stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the above-described method for large model output drift detection and repair based on historical statistics. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in an electronic device, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0098] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0100] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0102] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for detecting and repairing output drift of a large model based on historical statistics, characterized in that, include: S1: In the initial stage after model deployment, collect historical output samples of the large model under the preset prompt word set, and construct the historical statistical distribution baseline of each output dimension; S2: During the model running phase, collect the current output samples at a preset sampling period and extract the feature values ​​of the current output in each statistical dimension; S3: Based on the historical statistical distribution baseline, calculate the drift metric of the current output feature value, and determine whether output drift has occurred based on the comparison result of the drift metric and the dynamic threshold. S4: When output drift is detected, the drift type is identified in the current output to determine the category to which the drift belongs; S5: Based on the drift type, match the corresponding repair action from the preset repair strategy library and perform the repair operation; S6: Feed the repaired output back to the output buffer, write the repair record to the historical statistics database, and update the historical statistics distribution baseline.

2. The method according to claim 1, characterized in that, The construction of historical statistical distribution baselines for each output dimension in step S1 includes: S11: Determine the output statistical dimensions that need to be monitored. The statistical dimensions include at least one or more of the following: output length, lexical diversity, punctuation usage frequency, sentence structure distribution, sentiment polarity distribution, and frequency of occurrence of keywords in a specific domain. S12: For each statistical dimension, collect the statistical values ​​of a preset number of historical output samples to form a historical statistical value sequence for that dimension; S13: Perform a normality test on the historical statistical value series for each dimension. For dimensions that conform to a normal distribution, calculate their mean and standard deviation as baseline parameters. For dimensions that do not conform to a normal distribution, use quantile regression or kernel density estimation to establish a nonparametric statistical baseline. S14: Combine and store the baseline parameters of each dimension to form the historical statistical distribution baseline.

3. The method according to claim 1, characterized in that, Step S2, which involves extracting the feature values ​​of the current output across various statistical dimensions, includes: S21: Perform sentence segmentation and word segmentation preprocessing on the current output text to obtain a structured text unit sequence; S22: Traverse the sequence of text units and calculate the feature statistics of the current output in each statistical dimension; S23: Arrange the feature statistics of each dimension according to the same dimensions and order as the historical statistical distribution baseline to form the current output feature vector.

4. The method according to claim 1, characterized in that, Step S3, which involves calculating the drift metric of the current output feature value, includes: S31: For each statistical dimension, calculate the deviation between the current feature value and the baseline, based on the baseline type of that dimension; S32: Normalize the deviation values ​​to obtain standardized deviation scores for each dimension; S33: Weighted aggregation of the standardized deviation scores of each dimension to obtain the comprehensive drift metric of the current output; S34: The dynamic threshold is determined by calculating the dynamic threshold based on the dispersion of the historical statistical distribution baseline and a preset tolerance coefficient. The dynamic threshold is adaptively adjusted as the baseline is updated.

5. The method according to claim 1, characterized in that, The determination of whether output drift has occurred in step S3 is specifically as follows: When the overall drift metric exceeds the dynamic threshold, it is determined that the current output has drifted; otherwise, it is determined that the current output has not drifted. When the comprehensive drift metric value of a consecutive preset number of sampling periods does not exceed the dynamic threshold, the current output sample is included in the historical statistical database, triggering an incremental update of the historical statistical distribution baseline.

6. The method according to claim 1, characterized in that, The drift type identification in step S4 includes: S41: Extract the deviation direction and magnitude of the current output in each statistical dimension to form a drift feature vector; S42: Input the drift feature vector into a pre-trained drift classifier, which is trained based on historical drift samples and outputs the probability distribution of the drift belonging to each category; S43: Based on the probability distribution, the category with the highest probability is determined as the type of the current drift; The categories of drift include at least: semantic drift, style drift, factual drift, and format drift.

7. The method according to claim 1, characterized in that, The preset repair strategy library mentioned in step S5 contains repair actions corresponding to each drift type, including: For semantic drift, the corresponding repair action is to adjust the model decoding temperature parameter and / or top_p sampling parameter, and regenerate the output; For style drift, the corresponding fix is ​​to insert a style constraint instruction into the system prompt and regenerate the output. For factual drift, the corresponding remedial action is to trigger an external knowledge base search, inject the search results as contextual enhancement information into the prompt words, and regenerate the output; For format drift, the corresponding repair action is to apply a format normalization filter during the post-processing stage to correct the format of the output.

8. The method according to claim 1, characterized in that, After performing the repair operation as described in step S5, the repair effect is also verified: S51: Perform drift detection on the repaired output again and calculate the comprehensive drift metric of the repaired output; S52: If the comprehensive drift metric value of the repaired output is lower than the dynamic threshold, the repair is determined to be successful, and the repaired output is taken as the final output. S53: If the comprehensive drift metric value output after the repair is still not lower than the dynamic threshold, the repair is determined to have failed, and a backup repair strategy is triggered or a manual intervention alarm signal is generated.

9. The method according to claim 1, characterized in that, The step S6 of updating the historical statistical distribution baseline includes: S61: Store the output after a successful repair, along with its corresponding original prompt, repair action type, and repair timestamp, in the historical statistics database. S62: Recalculate the statistical baseline parameters for each dimension based on all historical samples in the historical statistical database according to the preset baseline update cycle; S63: Replace the original baseline parameters with the recalculated baseline parameters to complete the periodic update of the baseline.

10. A system for detecting and repairing large model output drift based on historical statistics, characterized in that, include: The output acquisition module is used to acquire the current output samples of the large model; The statistical feature extraction module is used to extract the feature values ​​of the output sample in each statistical dimension; The baseline storage module is used to store historical statistical distribution baselines constructed based on historical output samples; The drift detection module is used to calculate the drift metric based on the historical statistical distribution baseline and the current output feature value, and to determine whether output drift has occurred. The drift type identification module is used to identify the category to which the drift belongs when output drift occurs; The repair execution module is used to match the corresponding repair action from the preset repair strategy library according to the drift type and perform the repair operation; The baseline update module is used to update the historical statistical distribution baseline based on the output after successful repair and historical samples; The processor is communicatively connected to the output acquisition module, the statistical feature extraction module, the baseline storage module, the drift detection module, the drift type identification module, the repair execution module, and the baseline update module, respectively, and the processor is configured to execute the method as described in any one of claims 1 to 9.

Images