Adaptive threshold detection method and system for multi-dimensional distribution offset

Abstract

The invention discloses a multi-dimensional distribution offset adaptive threshold detection method and system, and the method comprises the steps: obtaining real-time data, extracting a multi-dimensional statistical feature, and obtaining a feature vector; based on historical normal data, using an expectation maximization algorithm to train a Gaussian mixture model, and determining parameters to obtain a normal distribution model; inputting the feature vector into the model, and calculating a probability value of the feature vector belonging to normal distribution as a first offset judgment index; based on the real-time data distribution of a plurality of detection objects in the same group, the distribution difference of any two objects is calculated by using a Wasserstein distance, and the similarity between the objects is obtained; and constructing a similarity network and calculating connectivity as a second offset judgment index. Setting a fixed-length sliding window, dynamically updating two indexes in the window, and obtaining a first self-adaptive threshold value and a second self-adaptive threshold value; and when any index is lower than a corresponding threshold value, determining distribution offset and giving an alarm, and updating model parameters in real time by using an incremental expectation maximization algorithm. According to the invention, accurate detection and intelligent analysis of data distribution offset are realized.

Description

Technical Field

[0001] This invention relates to the field of data distribution monitoring and anomaly detection technology, specifically to an adaptive threshold detection method and system for multi-dimensional distribution shift. Background Technology

[0002] With the rapid development of big data technology, data distribution monitoring is playing an increasingly important role in fields such as financial risk control, industrial quality management, and internet business monitoring. However, existing distribution offset detection technologies still have many shortcomings.

[0003] Traditional threshold methods suffer from static limitations. Existing technologies rely on historical data to set fixed thresholds. During periods of business fluctuation, such as e-commerce promotions or peak industrial seasons, frequent manual adjustments are required to adapt to changes in distribution. This is not only labor-intensive but also results in a false negative rate exceeding 30% due to adjustment delays, making it difficult to meet real-time detection needs.

[0004] Single-dimensional detection algorithms lack sufficient coverage. Mainstream solutions only model individual objects, ignoring group relationships among objects of the same category or product, and cannot identify systematic distribution anomalies (such as data synchronization shifts among multiple products of the same category), resulting in a lack of comprehensive risk warnings.

[0005] Static models and functional limitations restrict practicality. Most models are statically deployed after offline training, requiring additional computing power for periodic retraining; moreover, most technologies can only detect deviations and lack the ability to automatically locate root causes and predict trends at multiple granularities, making it difficult to support business decisions.

[0006] In summary, traditional distribution shift detection technologies mainly suffer from the following technical problems: First, traditional threshold methods cannot adapt to business changes and require constant manual adjustments, leading to detection delays and increased labor costs; second, traditional algorithms can only verify the distribution of single product values, ignoring group relationships and failing to identify systemic anomalies; third, static models are trained offline and cannot cope with data concept drift, resulting in performance degradation over time; fourth, most technologies lack the ability to perform shift root cause analysis and multi-granularity trend prediction, making it difficult to meet the needs of real-time, accurate, and comprehensive detection. Summary of the Invention

[0007] The purpose of this invention is to solve the above-mentioned technical problems existing in the prior art and to provide an adaptive threshold detection method and system for multi-dimensional distribution offset.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: This invention provides an adaptive threshold detection method for multi-dimensional distribution offset, comprising: Acquire real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors; Based on historical normal data, the Gaussian mixture model is trained using the expectation-maximization algorithm, the parameters of the Gaussian mixture model are determined, and the normal distribution model is obtained. The feature vector is input into the normal distribution model, and the probability value of the feature vector belonging to the normal distribution is calculated to obtain the first offset judgment index. Based on the real-time data distribution of multiple detected objects within the same group, the Wasserstein distance is used to calculate the distribution difference between any two detected objects to obtain the similarity between objects. A similarity network is constructed based on the similarity between the objects, and the connectivity of the similarity network is calculated to obtain a second offset determination index; wherein, the similarity network uses the detected object as a node and establishes edges between object pairs with a similarity greater than a preset similarity threshold; A fixed-length sliding window is set, and the first offset judgment index and the second offset judgment index within the sliding window are dynamically updated using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold. When the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold, a distribution offset is determined to have occurred, and a distribution offset alarm is output. Based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm to obtain the updated normal distribution model.

[0009] Furthermore, the step of extracting multi-dimensional statistical features from the real-time data to obtain a feature vector includes: Based on the real-time data, the arithmetic mean of the dataset is calculated to obtain the mean characteristic; Based on the real-time data and the mean feature, the degree of dispersion of the data deviating from the mean feature is calculated to obtain the standard deviation feature; Based on the real-time data and the mean characteristic, the degree of asymmetry in the data distribution is calculated to obtain the skewness characteristic; Based on the real-time data and the mean characteristic, the kurtosis of the data distribution is calculated to obtain the kurtosis characteristic; The real-time data is sorted, and the 25th percentile, 50th percentile, and 75th percentile are calculated to obtain the quartile features; The mean feature, standard deviation feature, skewness feature, kurtosis feature, and quartile feature are combined to obtain the feature vector.

[0010] Furthermore, the step of training a Gaussian mixture model using the expectation-maximization algorithm based on historical normal data, determining the parameters of the Gaussian mixture model, and obtaining a normal distribution model includes: Based on the historical normal data, initialize the number of sub-distributions, the initial weights of each sub-distribution, the initial mean vector, and the initial covariance matrix of the Gaussian mixture model. Execute the expectation step to calculate the posterior probability of each data point in the historical normal data belonging to each sub-distribution, and obtain the membership matrix. Based on the membership matrix and the historical normal data, a maximization step is performed to update the weights, mean vectors, and covariance matrices of each sub-distribution, resulting in updated Gaussian mixture model parameters. Based on the updated Gaussian mixture model parameters, the log-likelihood function value of the historical normal data is calculated, and it is determined whether the log-likelihood function value has converged, thus obtaining the convergence determination result. When the convergence determination result is non-convergence, the expectation step and the maximization step are repeatedly executed based on the updated Gaussian mixture model parameters until the log-likelihood function value converges, and the final Gaussian mixture model parameters are obtained. The parameters of the final Gaussian mixture model are used as the model parameters of the normal distribution model to obtain the normal distribution model.

[0011] Further, the step of inputting the feature vector into the normal distribution model, calculating the probability value of the feature vector belonging to the normal distribution, and obtaining the first offset determination index includes: Obtain the weights, mean vectors, and covariance matrices of each sub-distribution from the normal distribution model, substitute the feature vectors into the probability density functions of each sub-distribution, and calculate the probability density values ​​of the feature vectors under each sub-distribution. The probability density values ​​under each sub-distribution are multiplied by the weights of the corresponding sub-distributions and then summed to obtain the total probability value of the feature vector belonging to the normal distribution model; The total probability value is used as the first offset determination index to obtain the first offset determination index.

[0012] Furthermore, the method of calculating the distribution difference between any two detected objects based on the real-time data distribution of multiple detected objects within the same group, and obtaining the object similarity, includes: Obtain the real-time data distribution of the first detection object and the real-time data distribution of the second detection object, represent the real-time data distribution of the first detection object as a first probability distribution, and represent the real-time data distribution of the second detection object as a second probability distribution to obtain the first probability distribution and the second probability distribution; Based on the first probability distribution and the second probability distribution, the minimum transmission cost to transform the first probability distribution into the second probability distribution is calculated, and the Wasserstein distance is obtained. Based on the Wasserstein distance, the object similarity between the first detected object and the second detected object is calculated using the formula S=1 / (1+W).

[0013] Further, the step of constructing a similarity network based on the similarity between the objects, calculating the connectivity of the similarity network, and obtaining the second offset determination index includes: Each detected object within the same group is set as a node in the similarity network to obtain a node set; Traverse any two nodes in the node set. When the similarity between the objects corresponding to the two nodes is greater than the preset similarity threshold, establish an edge between the two nodes to obtain the similarity network. Based on the similarity network, a graph theory algorithm is used to identify connected clusters in the similarity network, and the connectivity is obtained by calculating the proportion of the number of nodes in the largest connected cluster to the total number of nodes in the node set. The connectivity is used as the second offset determination index to obtain the second offset determination index.

[0014] Further, the step of dynamically updating the first offset determination index and the second offset determination index within the sliding window using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold includes: Obtain the first offset determination index at the current time and the first adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the first adaptive threshold at the current time. Obtain the second offset determination index at the current time and the second adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the second adaptive threshold at the current time.

[0015] Furthermore, the duration of the sliding window is 24 hours.

[0016] Further, the step of determining a distribution offset and outputting a distribution offset alarm when the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold includes: Obtain the first offset determination index and the first adaptive threshold, determine whether the first offset determination index is lower than the first adaptive threshold, and obtain the first determination result; Obtain the second offset determination index and the second adaptive threshold, determine whether the second offset determination index is lower than the second adaptive threshold, and obtain the second determination result; When the first determination result is true or the second determination result is true, the current detection object is marked as having a distribution shift, and the distribution shift alarm is generated.

[0017] Furthermore, based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm to obtain the updated normal distribution model, including: Data that are both higher than the first adaptive threshold and higher than the second adaptive threshold in the sliding window are selected to obtain incremental training data. Based on the incremental training data and the parameters of the current Gaussian mixture model, an incremental expectation step is performed to calculate the posterior probability of each data point in the incremental training data belonging to each sub-distribution, thereby obtaining the incremental membership matrix. Based on the incremental membership matrix and the incremental training data, the incremental maximization step is performed to update the weights, mean vectors and covariance matrices of each sub-distribution, and to obtain candidate update parameters. Based on the candidate update parameters, calculate the rate of change of the mean vector in the candidate update parameters relative to the mean vector in the parameters of the current Gaussian mixture model, determine whether the rate of change is less than or equal to 10%, and obtain the stability determination result. When the stability determination result is true, the candidate update parameters are used to replace the parameters of the current Gaussian mixture model to obtain the updated normal distribution model.

[0018] Furthermore, it also includes: Obtain business logic rules and historical offset event data, and construct a directed acyclic graph based on the business logic rules and historical offset event data to obtain a causal graph; wherein, the result nodes of the causal graph are the first offset judgment index and the second offset judgment index, and the influencing factor nodes are business variables; When the distribution offset alarm is output, all influencing factor nodes are obtained from the causal graph, an intervention operation is performed on each influencing factor node, the value of the influencing factor node is fixed, and the change range of the first offset judgment index and the second offset judgment index in the causal graph after the intervention is calculated. Based on the magnitude of change, all the influencing factor nodes are sorted in descending order, and the top 3 influencing factor nodes with the largest magnitude of change are selected to obtain the key root causes; The key root causes are output to a visualization interface to obtain the root cause analysis results.

[0019] Furthermore, the construction of a directed acyclic graph based on the business logic rules and the historical offset event data to obtain a causal graph includes: Based on the aforementioned business logic rules, define the set of influencing factor variables and the set of result variables to obtain the initial node set; Based on the historical offset event data, a causal discovery algorithm is used to calculate the conditional independence between the influencing factor variables in the influencing factor variable set and the outcome variables in the outcome variable set, determine the direction of the causal edge, and obtain the causal edge set; The initial set of nodes is used as nodes, and the set of causal edges is used as directed edges to construct a directed acyclic graph, thus obtaining the causal graph.

[0020] Furthermore, it also includes: Set hourly, daily, and weekly time windows, wherein the length of the hourly time window is 12 hours, the length of the daily time window is 7 days, and the length of the weekly time window is 4 weeks, to obtain a multi-scale time window; For each of the multi-scale time windows, extract the time series data of the first offset determination index and the second offset determination index within the window, as well as the rate of change of statistical features, to obtain a time series feature set; Based on the historical time series feature set and the corresponding distribution offset label, a long short-term memory network model is used to learn the mapping relationship between the time series feature set and the distribution offset label, so as to obtain the trained trend prediction model. Input the current time series feature set into the trained trend prediction model, and output the predicted probability value of the distribution shift occurring in the future time period; Based on the predicted probability value, a high-risk warning is issued when the predicted probability value is greater than 80%, a medium-risk warning is issued when the predicted probability value is between 50% and 80%, and a low-risk warning is issued when the predicted probability value is less than 50%, thus obtaining a graded warning result.

[0021] Further, the step of learning the mapping relationship between the time-series feature set and the corresponding distribution offset labels using a long short-term memory network model based on the historical time-series feature set and the corresponding distribution offset labels to obtain the trained trend prediction model includes: A long short-term memory network model architecture is constructed, which includes an input layer, a long short-term memory layer, a fully connected layer, and an output layer, to obtain an initial trend prediction model; Using the historical time series feature set as input and the distribution offset label as a supervision signal, the weight parameters of the initial trend prediction model are trained using the backpropagation algorithm. The cross-entropy loss between the predicted probability value output by the initial trend prediction model and the distribution offset label is minimized to obtain the training process loss value. When the loss value of the training process converges, the weight parameters of the initial trend prediction model are saved to obtain the completed trend prediction model.

[0022] The present invention also provides an adaptive threshold detection system for multi-dimensional distribution offset, comprising: The feature extraction module is used to acquire real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors. The normal distribution model training module is used to train a Gaussian mixture model based on historical normal data using the expectation-maximization algorithm, determine the parameters of the Gaussian mixture model, and obtain the normal distribution model. The first offset determination module is used to input the feature vector into the normal distribution model, calculate the probability value of the feature vector belonging to the normal distribution, and obtain the first offset determination index. The inter-object similarity calculation module is used to calculate the distribution difference between any two detected objects based on the real-time data distribution of multiple detected objects within the same group, and obtain the inter-object similarity. The second offset determination module is used to construct a similarity network based on the similarity between the objects, calculate the connectivity of the similarity network, and obtain a second offset determination index; wherein, the similarity network uses the detected object as a node and establishes edges between object pairs with a similarity greater than a preset similarity threshold; An adaptive threshold update module is used to set a sliding window of fixed length and dynamically update the first offset judgment index and the second offset judgment index within the sliding window using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold. The distribution offset determination module is used to determine that a distribution offset has occurred and output a distribution offset alarm when the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold. The model incremental update module is used to update the parameters of the Gaussian mixture model in real time based on the normal data within the sliding window using the incremental expectation-maximization algorithm, so as to obtain the updated normal distribution model.

[0023] The beneficial effects of this invention are: 1. By combining multi-dimensional GMM modeling with group detection, we break through the limitations of traditional single-dimensional methods. GMM can fit non-normal business data and accurately characterize individual distribution features. Combined with Wasserstein distance and similarity networks, it can identify group consistency anomalies (such as collective distribution shifts of products in the same category), which reduces misjudgments due to individual random fluctuations and avoids missing systemic risks, thereby improving detection coverage. 2. Fully automatic adaptation is achieved through adaptive thresholds and incremental learning. The EWMA dynamic threshold automatically adjusts with business fluctuations (such as major promotions and seasonal changes) without manual intervention; incremental learning updates GMM parameters in real time to cope with conceptual drift such as changes in user habits, resulting in good long-term model accuracy and improved alarm timeliness. 3. Fill explanatory gaps through causal root cause analysis. Key factors causing distribution deviations (such as promotional cessation or price adjustments) can be identified quickly, replacing manual investigation, shortening problem-solving cycles, and providing a basis for rapid loss mitigation in business operations; 4. Achieve proactive prevention through multi-scale forecasting. Hourly, daily, and weekly early warnings cover short, medium, and long-term demand, predicting distribution changes 1-14 days in advance, assisting in decisions such as inventory adjustments and process optimization, and driving risk response from "passive response" to "proactive prevention". Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the adaptive threshold detection method for multi-dimensional distribution offset of the present invention. Figure 2 This is a structural framework diagram of the adaptive threshold detection system with multi-dimensional distribution offset of the present invention. Detailed Implementation

[0026] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0027] Example 1: like Figure 1 As shown, this invention provides an adaptive threshold detection method for multi-dimensional distribution offset, comprising: Real-time data is acquired, and multi-dimensional statistical features are extracted from the real-time data to obtain feature vectors. Based on historical normal data, a Gaussian mixture model is trained using the expectation-maximization algorithm to determine the parameters of the Gaussian mixture model, resulting in a normal distribution model. The feature vectors are input into the normal distribution model, and the probability value of the feature vectors belonging to the normal distribution is calculated to obtain a first offset judgment index. Based on the real-time data distribution of multiple detection objects within the same group, the distribution difference between any two detection objects is calculated using Wasserstein distance to obtain object similarity. Based on the object similarity, a similarity network is constructed, and the connectivity of the similarity network is calculated to obtain a second offset judgment index. Wherein, the... The similarity network uses the detected objects as nodes and establishes edges between object pairs with a similarity greater than a preset similarity threshold. A fixed-length sliding window is set, and the first offset judgment index and the second offset judgment index within the sliding window are dynamically updated using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold. When the first offset judgment index is lower than the first adaptive threshold or the second offset judgment index is lower than the second adaptive threshold, a distribution offset is determined to have occurred, and a distribution offset alarm is output. Based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using an incremental expectation-maximization algorithm to obtain an updated normal distribution model.

[0028] S1. Obtain real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors.

[0029] First, real-time data streams are acquired from business systems. This real-time data can come from various sources, such as transaction records from e-commerce platforms, sensor data from industrial production lines, and transaction data from financial systems. Real-time data acquisition employs streaming processing to ensure the timeliness and continuity of the data. After acquiring the raw real-time data, the system preprocesses the data, including data cleaning, missing value handling, and preliminary outlier filtering, to ensure the accuracy of subsequent feature extraction.

[0030] After data preprocessing, the system extracts multi-dimensional statistical features from the real-time data. These multi-dimensional statistical features are multiple statistical indicators extracted from the dataset that comprehensively characterize the data distribution, including mean, standard deviation, skewness, kurtosis, and quartiles. The mean reflects the central tendency of the data, the standard deviation reflects the dispersion, the skewness reflects the symmetry of the data distribution, the kurtosis reflects the sharpness of the data distribution, and the quartiles reflect the quantile information of the data. By extracting these multi-dimensional features, the system can comprehensively describe the distribution characteristics of the data from different perspectives, avoiding the bias caused by a single feature. After extraction, the system combines these statistical features into a feature vector, which will serve as input for subsequent distribution modeling and offset determination.

[0031] S2. Based on historical normal data, train a Gaussian mixture model using the expectation-maximization algorithm, determine the parameters of the Gaussian mixture model, and obtain the normal distribution model. A Gaussian mixture model is trained using the expectation-maximization algorithm based on historical normal data. Historical normal data refers to data accumulated during normal system operation, which has been manually labeled or automatically filtered to confirm its conformity to normal business patterns. A Gaussian mixture model is a probabilistic model that assumes the data is a mixture of multiple Gaussian distributions (also known as normal distributions), each called a sub-distribution or component. Gaussian mixture models can fit complex non-normal distributions and are particularly suitable for handling business data distributions with multimodal or asymmetric characteristics.

[0032] The Expectation-Maximization (EM) algorithm is an iterative optimization algorithm used to estimate parameters in probabilistic models containing latent variables. The basic principle of this algorithm is to alternately execute expectation and maximization steps. In the expectation step, the posterior probabilities of the latent variables are estimated based on the current parameters. In the maximization step, the model parameters are updated based on the posterior probabilities of the latent variables. Through multiple iterations, the model parameters gradually converge to their optimal values. The system uses the Expectation-Maximization algorithm to determine the parameters of a Gaussian mixture model, including the weights, mean vectors, and covariance matrices of each sub-distribution. The sub-distribution weights represent the proportion of each sub-distribution in the overall distribution, the mean vectors represent the center position of each sub-distribution, and the covariance matrix represents the shape and orientation of each sub-distribution.

[0033] The training process continues until the model parameters converge, meaning the change in the log-likelihood function value is less than a preset threshold. The resulting Gaussian mixture model after training is the normal distribution model, which can describe the distribution characteristics of normal business data and provide a benchmark for subsequent offset determination.

[0034] S3. Input the feature vector into the normal distribution model, calculate the probability value that the feature vector belongs to the normal distribution, and obtain the first offset judgment index. The feature vector extracted from real-time data is input into a pre-trained normal distribution model to calculate the probability that the feature vector belongs to the normal distribution. Specifically, the system obtains the parameters of each sub-distribution from the normal distribution model, including weights, mean vector, and covariance matrix. Then, it substitutes the feature vector into the probability density function of each sub-distribution for calculation. The probability density function describes the relative likelihood of a continuous random variable taking a certain value. For a Gaussian distribution, its probability density function is uniquely determined by the mean vector and covariance matrix.

[0035] After calculating the probability density values ​​of the feature vector under each sub-distribution, these probability density values ​​are multiplied by the weights of the corresponding sub-distributions. Then, all weighted probability density values ​​are summed to obtain the total probability value that the feature vector belongs to the normal distribution model. This total probability value reflects the similarity between the distribution characteristics of the current real-time data and the historical normal data distribution. A higher probability value indicates a closer fit to the normal distribution, while a lower probability value indicates a higher likelihood of distribution shift. The system uses this total probability value as the first shift determination criterion for subsequent shift assessments.

[0036] S4. Based on the real-time data distribution of multiple detected objects within the same group, the Wasserstein distance is used to calculate the distribution difference between any two detected objects to obtain the object similarity. Based on the real-time data distribution of multiple detected objects within the same group, the Wasserstein distance is used to calculate the distribution difference between any two detected objects. The same group refers to a set of objects with the same or similar business attributes, such as goods of the same category, equipment of the same model, or users of the same type. The Wasserstein distance, also known as the bulldozer distance, is a method for measuring the difference between two probability distributions. Its basic principle is the minimum cost required to transform one probability distribution into another. This can be figuratively understood as the minimum amount of work required to move a pile of soil from one distribution shape to another, where the amount of work equals the amount of soil moved multiplied by the distance moved.

[0037] Compared to other distribution distance metrics (such as KL divergence), Wasserstein distance has the advantage of being sensitive to non-overlapping distributions. Even if two distributions do not overlap at all, Wasserstein distance can still provide a meaningful metric. The system acquires the real-time data distributions of the first and second detection objects, representing them as a first probability distribution and a second probability distribution, respectively. Then, it calculates the minimum transmission cost to transform the first probability distribution into the second probability distribution, obtaining the Wasserstein distance. Based on the calculated Wasserstein distance, the system uses a conversion formula to convert the distance value into a similarity value. The converted similarity value is between 0 and 1; the closer the value is to 1, the more similar the distributions of the two objects are, and the closer the value is to 0, the greater the difference in distributions.

[0038] S5. Construct a similarity network based on the object similarity, calculate the connectivity of the similarity network, and obtain a second offset determination index; wherein, the similarity network uses the detected object as a node, and establishes edges between object pairs with a similarity greater than a preset similarity threshold. A similarity network is constructed based on the calculated object similarity. A similarity network is a graph structure where nodes represent detected objects and edges represent similarity relationships between objects. The system sets each detected object within the same group as a node in the similarity network, forming a node set. Then, the system iterates through any two nodes in the node set, calculating the similarity between their corresponding objects. When the similarity exceeds a preset similarity threshold, an edge is established between these two nodes. The preset similarity threshold is set based on business characteristics and historical experience, typically around 0.8, indicating that a connection is only established between object pairs with high similarity.

[0039] After constructing the similarity network, the network's connectivity is calculated. Connectivity is an indicator of the overall tightness of network connections, reflecting the consistency of object distribution within the group. The system uses graph theory algorithms to identify connected clusters in the similarity network. A connected cluster is a subset of nodes in the network that are interconnected; any two nodes within a connected cluster can be reached from each other via an edge. The system calculates the number of nodes in the largest connected cluster and the proportion of this number to the total number of nodes in the cluster, thus obtaining the connectivity. Under normal circumstances, the distribution of most objects within the group should be similar, and the similarity network will form a large connected cluster with high connectivity. When a group distribution shift occurs, connected clusters may split or more isolated nodes may appear, resulting in a decrease in connectivity. The calculated connectivity is used as a second shift criterion to detect anomalies in group consistency.

[0040] S6. Set a sliding window of fixed length, and use an exponentially weighted moving average algorithm to dynamically update the first offset judgment index and the second offset judgment index within the sliding window to obtain the first adaptive threshold and the second adaptive threshold.

[0041] A fixed-length sliding window is set up, and an exponentially weighted moving average algorithm is used to dynamically update the first and second offset judgment indicators within the sliding window, respectively, to obtain the first and second adaptive thresholds. The sliding window is a time series data processing method where the system sets a fixed-length time window, which slides forward over time, always maintaining data from the most recent period. The length of the sliding window is set according to business characteristics, typically 24 hours, to capture cyclical changes in business operations within a day.

[0042] The exponentially weighted moving average (ERM) algorithm is a time series smoothing method that assigns exponentially decaying weights to historical data, with more recent data receiving greater weight and earlier data receiving less weight. The core idea of ​​this algorithm is to use the weighted sum of the current value and the average of the previous time step as the new average value for the current time step, controlling the weight ratio between historical and current data through a smoothing coefficient. The system obtains the first offset judgment index for the current time step and the first adaptive threshold for the previous time step, sets the smoothing coefficient to 0.3, and calculates the first adaptive threshold for the current time step. A smoothing coefficient of 0.3 means that the current value accounts for 30% of the weight, and the historical average accounts for 70% of the weight. This setting allows for rapid response to data changes while avoiding frequent fluctuations caused by oversensitivity.

[0043] Similarly, by obtaining the second offset judgment index at the current moment and the second adaptive threshold at the previous moment, and using the same smoothing coefficient of 0.3, the second adaptive threshold at the current moment is calculated. Through the exponentially weighted moving average algorithm, the system achieves adaptive adjustment of the threshold. The threshold can dynamically change with the normal fluctuations of business data without manual intervention, effectively solving the problem that traditional fixed thresholds are difficult to adapt to business changes.

[0044] S7. When the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold, a distribution offset is determined to have occurred, and a distribution offset alarm is output.

[0045] The system compares the current first offset criterion with a first adaptive threshold, or the second offset criterion with a second adaptive threshold, to determine whether a distribution shift has occurred. Specifically, the system obtains the first offset criterion and the first adaptive threshold, determines whether the first offset criterion is lower than the first adaptive threshold, and obtains a first determination result. A first offset criterion lower than the threshold indicates that the distribution characteristics of the current data significantly deteriorate in relation to the normal distribution model, potentially indicating an individual-level distribution shift.

[0046] Simultaneously, a second offset judgment index and a second adaptive threshold are obtained. The system determines whether the second offset judgment index is lower than the second adaptive threshold, thus obtaining a second judgment result. A second offset judgment index below the threshold indicates a significant decrease in the connectivity of the similarity network, suggesting a reduction in the distribution consistency of objects within the group, potentially indicating a group-level distribution shift. The system uses "OR" logic to combine the two judgment results; that is, when either the first judgment result or the second judgment result is true, the system marks the currently detected object as having experienced a distribution shift and generates a distribution shift alarm. This dual judgment mechanism can detect both individual and group anomalies, improving the comprehensiveness and accuracy of offset detection.

[0047] S8. Based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm to obtain the updated normal distribution model.

[0048] Based on normal data within a sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm, resulting in an updated normal distribution model. Incremental learning is a model update method that does not require retraining the entire model; instead, it adjusts the existing model parameters based on new data, offering advantages such as high computational efficiency and fast response speed. First, normal data is selected from the sliding window. The selection criteria are that a first offset criterion is higher than a first adaptive threshold and a second offset criterion is higher than a second adaptive threshold. Data meeting both conditions is considered normal data and can be used for model updates.

[0049] Based on the selected incremental training data and the parameters of the current Gaussian mixture model, an incremental expectation step is performed to calculate the posterior probability of each data point in the incremental training data belonging to each sub-distribution, resulting in an incremental membership matrix. Then, based on the incremental membership matrix and the incremental training data, an incremental maximization step is performed to update the weights, mean vectors, and covariance matrices of each sub-distribution, yielding candidate update parameters. To ensure the stability of the model update and avoid model failure due to drastic parameter changes, the rate of change of the mean vector in the candidate update parameters relative to the mean vector in the current model parameters is calculated. The rate of change is then checked to determine if it is less than or equal to 10%, resulting in a stability assessment. Only when the stability assessment result is true are the candidate update parameters used to replace the current Gaussian mixture model parameters, completing the model update. Through this incremental learning mechanism, the model can continuously adapt to the slow evolution of business data, effectively address the concept drift problem, and maintain long-term detection accuracy.

[0050] After completing a full detection process, the system returns to step S1 to acquire new real-time data and repeat the steps of feature extraction, offset determination, threshold update, and model update, forming a closed loop of continuous monitoring. The entire detection system operates in a streaming mode, responding to data changes in real time and providing continuous distributed offset monitoring services to business systems.

[0051] Example 2: In this embodiment, based on the real-time data, the arithmetic mean of the dataset is calculated to obtain the mean feature; based on the real-time data and the mean feature, the dispersion of the data deviating from the mean feature is calculated to obtain the standard deviation feature; based on the real-time data and the mean feature, the asymmetry of the data distribution is calculated to obtain the skewness feature; based on the real-time data and the mean feature, the kurtosis of the data distribution is calculated to obtain the kurtosis feature; the real-time data is sorted, and the 25th percentile, 50th percentile, and 75th percentile are calculated to obtain the quartile feature; the mean feature, the standard deviation feature, the skewness feature, the kurtosis feature, and the quartile feature are combined to obtain the feature vector.

[0052] Specifically, firstly, based on the acquired real-time data, the arithmetic mean of the dataset is calculated to obtain the mean feature. The arithmetic mean is the most basic measure of central tendency; it sums all data values ​​and divides by the number of data points, reflecting the average level or central location of the data. The mean feature describes the overall level of the data and is a fundamental characteristic for characterizing data distribution. In practical applications, the mean feature can reflect the average state of business indicators, such as the average price of goods, the average operating parameters of equipment, or the average spending of users. The system saves the calculated arithmetic mean as the mean feature for subsequent feature vector construction.

[0053] Secondly, based on the real-time data and the mean characteristic, the degree of dispersion of the data deviating from the mean characteristic is calculated to obtain the standard deviation characteristic. Standard deviation is a commonly used indicator to measure the dispersion of data; it measures the average deviation of a data point from the mean. The calculation process for standard deviation is as follows: first, calculate the difference between each data point and the mean; then, square the difference to eliminate the influence of the sign; next, average all squared differences; finally, take the square root to obtain the standard deviation. A larger standard deviation indicates greater data fluctuation and a more dispersed distribution; a smaller standard deviation indicates that the data is more concentrated around the mean and the distribution is more concentrated.

[0054] In distribution offset detection, the standard deviation feature reflects changes in data volatility. When business anomalies occur, the standard deviation of the data may increase or decrease significantly. For example, increased price volatility, unstable equipment operation, or dispersed user behavior can all lead to changes in the standard deviation. The calculated standard deviation is used as the standard deviation feature to describe the discrete characteristics of the data.

[0055] Next, based on the real-time data and the mean characteristics, the degree of asymmetry in the data distribution is calculated to obtain the skewness characteristic. Skewness is a statistical indicator that measures the symmetry of a data distribution; it describes the direction and degree of skewness of the data distribution relative to the mean. Skewness is calculated based on the cube of the difference between a data point and the mean, and a dimensionless skewness coefficient is obtained through standardization. A skewness of zero indicates a perfectly symmetrical distribution, meaning the data is distributed equally on both sides of the mean, exhibiting a symmetrical bell-shaped curve. A positive skewness indicates a right-skewed distribution, meaning the tail on the right side of the data is longer than the tail on the left, with more extreme values ​​above the mean. A negative skewness indicates a left-skewed distribution, meaning the tail on the left side of the data is longer than the tail on the right, with more extreme values ​​below the mean.

[0056] In practical business operations, skewness features can capture changes in the distribution pattern of data. For example, under normal circumstances, user spending may exhibit a right-skewed distribution, with most users spending relatively little and a few spending very much. When promotional activities occur or user behavior changes, the skewness of the distribution may change, which can then be detected by the system. The calculated skewness coefficient is used as a skewness feature to characterize the asymmetry of the distribution.

[0057] Then, based on the real-time data and the mean characteristic, the kurtosis of the data distribution is calculated to obtain the kurtosis characteristic. Kurtosis is a statistical indicator that measures the shape of a data distribution; it describes the sharpness of the top and the thickness of the tails of the distribution curve. Kurtosis is calculated based on the fourth power of the difference between the data point and the mean, and the kurtosis coefficient is obtained through standardization. The kurtosis of the standard normal distribution is 3. Usually, 3 is subtracted from the kurtosis to obtain the excess kurtosis, making the excess kurtosis of the normal distribution zero for easier comparison. A positive excess kurtosis indicates that the distribution is sharper than the normal distribution, with a more prominent top and thicker tails; this is called a leptokurtic distribution or a heavy-tailed distribution. A negative excess kurtosis indicates that the distribution is flatter than the normal distribution, with a flatter top and thinner tails; this is called a flat-peaked distribution or a thin-tailed distribution.

[0058] Kurtosis is important in anomaly detection because it reflects the frequency of extreme values. A heavy-tailed distribution indicates a higher probability of extreme values, which is a significant warning signal in scenarios such as financial risk and industrial failures. A significant change in the kurtosis of a data distribution may indicate a change in the system's operating mode, requiring attention. The calculated kurtosis coefficient is used as the kurtosis feature to describe the sharpness of the distribution.

[0059] Next, the real-time data is sorted, and the 25th, 50th, and 75th percentiles are calculated to obtain the quartile characteristics. A percentile is the value at a specific percentage position after data is arranged in ascending order. The 25th percentile, also known as the first quartile, indicates that 25% of the data is less than or equal to this value, and 75% of the data is greater than this value. The 50th percentile, also known as the median or second quartile, indicates that 50% of the data is less than or equal to this value, and 50% of the data is greater than this value; it is the center position of the data. The 75th percentile, also known as the third quartile, indicates that 75% of the data is less than or equal to this value, and 25% of the data is greater than this value.

[0060] Compared to mean features, quartile features are less sensitive to extreme values ​​and can more stably reflect the distribution of data. The intervals between quartiles reflect the dispersion of data; the difference between the third quartile and the first quartile is called the interquartile range, a robust indicator of data volatility. In distribution shift detection, changes in quartile features can reflect the shift in the center and range of the data distribution, especially in the presence of extreme value interference, where quartile features are more reliable than the mean and standard deviation. The calculated 25th, 50th, and 75th percentiles are used as quartile features to supplement the description of the data's distribution characteristics.

[0061] Finally, the mean feature, standard deviation feature, skewness feature, kurtosis feature, and quartile feature are combined to obtain the feature vector.

[0062] An eigenvector is a vector formed by arranging multiple eigenvalues ​​in a certain order. It is a point in a multidimensional space and can comprehensively characterize the distribution characteristics of data. Five types of features are combined into a multidimensional eigenvector, specifically including one mean, one standard deviation, one skewness, one kurtosis, and three quartiles, forming a 7-dimensional eigenvector. This eigenvector characterizes the distribution characteristics of real-time data from multiple perspectives, such as central tendency, dispersion, distribution pattern, and quantile information. It provides rich input information for subsequent Gaussian mixture modeling and can more accurately identify distribution shifts.

[0063] Example 3: In this embodiment, based on the historical normal data, the number of sub-distributions, the initial weights, the initial mean vector, and the initial covariance matrix of the Gaussian mixture model are initialized. An expectation step is executed to calculate the posterior probability of each data point in the historical normal data belonging to each sub-distribution, resulting in a membership matrix. Based on the membership matrix and the historical normal data, a maximization step is executed to update the weights, mean vectors, and covariance matrices of each sub-distribution, resulting in updated Gaussian mixture model parameters. Based on the updated Gaussian mixture model parameters, the log-likelihood function value of the historical normal data is calculated, and it is determined whether the log-likelihood function value has converged, resulting in a convergence determination result. If the convergence determination result is non-convergence, the expectation step and the maximization step are repeated based on the updated Gaussian mixture model parameters until the log-likelihood function value converges, resulting in the final Gaussian mixture model parameters. The final Gaussian mixture model parameters are used as the model parameters of the normal distribution model to obtain the normal distribution model.

[0064] Specifically, based on historical normal data, the parameters of the Gaussian mixture model are initialized. First, the number of sub-distributions is determined, i.e., how many Gaussian distribution components are included in the Gaussian mixture model. The choice of the number of sub-distributions needs to balance the model's expressive power and complexity. Too few sub-distributions may not adequately fit the complex distribution, while too many may lead to overfitting and computational burden. The system can determine the appropriate number of sub-distributions through cross-validation, information criteria (such as the BIC or AIC criterion), or domain knowledge. Typically, setting it to 3 to 5 sub-distributions can meet the needs of most business scenarios.

[0065] After determining the number of sub-distributions, the system initializes the initial weights, initial mean vectors, and initial covariance matrices for each sub-distribution. The initial weights represent the proportion of each sub-distribution in the mixture model. Typically, the weights are initialized equally; for example, with three sub-distributions, the initial weight of each sub-distribution is one-third, ensuring the sum of all weights is 1. The initial mean vector represents the center position of each sub-distribution. The system can use the K-means clustering algorithm to cluster historical normal data, using the cluster centers as the initial mean vectors for each sub-distribution. This initialization method makes the initial parameters closer to the true distribution, accelerating convergence. The initial covariance matrix represents the shape and orientation of each sub-distribution. The covariance matrix can be initialized as a diagonal matrix, with the diagonal elements set to the average of the variances of each dimension of the data, indicating that each dimension is independent and has the same initial variance.

[0066] After parameter initialization is complete, the system is ready to enter the iterative optimization process, gradually adjusting the parameters through the expectation-maximization algorithm to make the model better fit the distribution of historical normal data.

[0067] Secondly, based on the current Gaussian mixture model parameters, an expectation step is performed to calculate the posterior probability of each data point in the historical normal data belonging to each sub-distribution. The core of the expectation step is to estimate the probability distribution of latent variables using the current parameters, based on Bayes' theorem. Here, latent variables refer to which sub-distribution each data point belongs to; this information is unknown in the observed data and needs to be estimated through probabilistic inference.

[0068] For each data point in the historical normal data, calculate the probability density value of that data point under each subdistribution, and then multiply it by the weight of the corresponding subdistribution to obtain the joint probability. Divide the joint probability corresponding to a certain subdistribution by the sum of the joint probabilities of all subdistributions to obtain the posterior probability of that data point belonging to that subdistribution, also known as the degree of responsibility or membership. The posterior probability reflects the likelihood that the data point comes from a certain subdistribution given that the data point has been observed.

[0069] Repeat the above calculation for all data points and all subdistributions in the historical normal data to obtain a membership matrix. Rows in the membership matrix correspond to data points, and columns correspond to subdistributions. Each element in the matrix represents the posterior probability that a data point belongs to a particular subdistribution. The sum of the elements in each row of the membership matrix is ​​1, indicating that each data point necessarily belongs to a certain subdistribution. The membership matrix is ​​the foundation for subsequent maximization steps, providing a soft assignment relationship between data points and subdistributions.

[0070] Next, based on the membership matrix calculated in the expectation step and the historical normal data, the maximization step is performed to update the weights, mean vectors, and covariance matrices of each subdistribution. The goal of the maximization step is to find the parameter values ​​that maximize the log-likelihood function of the data, given the membership matrix.

[0071] To update the weights of the sub-distributions, calculate the sum of the membership degrees of each sub-distribution, which is the sum of the elements in the corresponding columns of the membership matrix. Then divide this sum by the total number of data points to obtain the new weights. The new weights reflect the degree to which each sub-distribution interprets the data under the current parameters; the sub-distribution with the larger the sum of its membership degrees, the larger its weight.

[0072] To update the mean vector, for each subdistribution, sum the products of all data points and their membership degrees, then divide by the sum of the membership degrees of that subdistribution to obtain the new mean vector. This is equivalent to performing a weighted average of the data points, with the weights being the posterior probability of each data point belonging to that subdistribution, so that data points that contribute significantly to that subdistribution have a larger proportion in the mean calculation.

[0073] To update the covariance matrix, for each sub-distribution, calculate the difference vector between the data points and the new mean vector. Then, calculate the product of the difference vector and its transpose to obtain each element of the covariance matrix. Finally, perform membership weighting and normalization to obtain the new covariance matrix. The new covariance matrix reflects the distribution of data relative to the center of the sub-distribution under the current parameters.

[0074] After updating the weights, mean vector, and covariance matrix, the updated Gaussian mixture model parameters are obtained. Compared with the initial parameters or the parameters from the previous iteration, these new parameters can better fit the distribution of historical normal data.

[0075] Then, based on the updated Gaussian mixture model parameters, the log-likelihood function value of the historical normal data is calculated to determine whether the model has converged. The log-likelihood function is an indicator of how well a model fits the data; it represents the logarithm of the probability of observing the current data given the model parameters. The larger the log-likelihood function value, the stronger the model's interpretability of the data and the better the fit.

[0076] The system calculates the probability value of each data point under the current Gaussian mixture model, which is the weighted sum of the probability density values ​​of each sub-distribution. Then, it takes the logarithm of all data point probability values ​​and sums them to obtain the log-likelihood function value. The system compares the current iteration's log-likelihood function value with the previous iteration's log-likelihood function value, calculating the difference or relative rate of change between the two. If the difference is less than a preset convergence threshold (e.g., 0.001) or the relative rate of change is less than a preset percentage (e.g., 0.01%), the system determines that the log-likelihood function has converged, resulting in a true convergence result. This indicates that the model parameters have stabilized, and further iteration will not bring significant improvement. If the difference or relative rate of change is still greater than the threshold, the convergence result is false, indicating that the model still has room for optimization and further iteration is needed.

[0077] Finally, if the convergence determination result is non-convergence, based on the updated Gaussian mixture model parameters, return to step S2.2 and repeat the expectation step and maximization step. In a new iteration, the system recalculates the membership matrix using the just-updated parameters, and then updates the parameters again, repeating this cycle. Each iteration increases or keeps the log-likelihood function value unchanged, and the model gradually converges towards the optimal parameters.

[0078] The iteration continues until the log-likelihood function value converges, i.e., the convergence condition in step S2.4 is met. At this point, the final Gaussian mixture model parameters are obtained, which allow the model to achieve the optimal fit to historical normal data. To prevent infinite iteration, the system also sets a maximum iteration limit (e.g., 100 or 200 times). When the maximum iteration limit is reached, the iteration terminates even if it has not fully converged, and the current parameters are used as the final parameters.

[0079] The final Gaussian mixture model parameters obtained after convergence, including the number of sub-distributions, the weights of each sub-distribution, the mean vector, and the covariance matrix, are saved as the model parameters for the normal distribution model. This normal distribution model fully learns the distribution characteristics of historical normal data, including the multimodal nature of the data, the position and shape of each peak, and other complex characteristics, providing an accurate benchmark model for subsequent real-time data distribution shift determination.

[0080] Example 4: In this embodiment, the weights, mean vectors, and covariance matrices of each sub-distribution are obtained from the normal distribution model. The feature vector is substituted into the probability density function of each sub-distribution to calculate the probability density value of the feature vector under each sub-distribution. The probability density values ​​under each sub-distribution are multiplied by the weights of the corresponding sub-distributions and then summed to obtain the total probability value of the feature vector belonging to the normal distribution model. The total probability value is used as the first offset determination index to obtain the first offset determination index.

[0081] Specifically, the weights, mean vectors, and covariance matrices of each sub-distribution are first obtained from the trained normal distribution model. These parameters were optimized using the expectation-maximization algorithm during the training process in Example 3, and fully describe the structure of the Gaussian mixture model. The weight parameter is a vector whose length is equal to the number of sub-distributions. Each element represents the proportion of the corresponding sub-distribution in the mixture model, and the sum of all weights equals 1. The mean vector is a matrix, with each row corresponding to the mean vector of a sub-distribution. The dimension of the vector is equal to the dimension of the feature vector, which is 7 dimensions in this invention, corresponding to the mean feature, standard deviation feature, skewness feature, kurtosis feature, and three quartile features, respectively. The covariance matrix is ​​a three-dimensional tensor. There is a covariance matrix for each sub-distribution, with a size of 7×7, describing the shape, orientation, and diffusion degree of the sub-distribution in the 7-dimensional feature space.

[0082] After obtaining these model parameters, the feature vectors extracted from the real-time data are substituted into the probability density functions of each subdistribution for calculation. The probability density function describes the probability of a continuous random variable taking a value near a certain point. For a multidimensional Gaussian distribution, its probability density function is uniquely determined by the mean vector and the covariance matrix. The mathematical form of the probability density function includes an exponential term and normalization coefficients. The exponential term contains the difference between the feature vector and the mean vector, as well as the inverse of the covariance matrix. The normalization coefficients are related to the determinant of the covariance matrix.

[0083] For each subdistribution, the calculations are performed separately. First, the difference vector between the eigenvector and the mean vector of that subdistribution is calculated. Then, the inverse and determinant of the covariance matrix of that subdistribution are calculated. The inverse of the covariance matrix is ​​used to perform a linear transformation on the difference vector, measuring the Mahalanobis distance of the eigenvector from the mean. The determinant of the covariance matrix is ​​used for normalization, ensuring that the integral of the probability density function over the entire space is 1. Substituting the difference vector, the inverse covariance matrix, and the determinant into the formula for calculating the probability density function yields the probability density value of the eigenvector under that subdistribution.

[0084] This probability density value represents the relative likelihood of observing the current feature vector under the assumptions of that sub-distribution. A larger probability density value indicates that the current feature vector more closely matches the characteristics of that sub-distribution and is more likely to originate from it. A smaller probability density value indicates that the current feature vector differs significantly from the characteristics of that sub-distribution and is less likely to originate from it. Repeating the above calculation process for all sub-distributions yields the set of probability density values ​​for the feature vector under each sub-distribution.

[0085] Next, the probability density values ​​under each sub-distribution are multiplied by their corresponding weights and summed to obtain the total probability value of the eigenvector belonging to the normal distribution model. This calculation process embodies the core idea of ​​the Gaussian mixture model, that is, the total probability is a weighted combination of the probabilities of each sub-distribution, and the weights reflect the degree of contribution of each sub-distribution to the overall distribution.

[0086] In the specific calculation, the probability density value of the first sub-distribution is taken and multiplied by its weight to obtain the first weighted probability density value. Similarly, the system calculates the weighted probability density values ​​of the second sub-distribution, the third sub-distribution, and so on, until the weighted probability density values ​​of all sub-distributions have been calculated. Then, all weighted probability density values ​​are summed to obtain the total probability value.

[0087] This overall probability value comprehensively considers the performance of the eigenvector under each subdistribution and the importance of each subdistribution in the normal distribution model. If the eigenvector has a high probability density value under a subdistribution with a large weight, even if the probability density values ​​are lower under other subdistributions, the overall probability value will still be high, indicating that the eigenvector conforms to the main pattern of the normal distribution model. Conversely, if the eigenvector has low probability density values ​​under all subdistributions, or only high values ​​under subdistributions with very small weights, the overall probability value will be low, indicating that the eigenvector deviates from the distribution characteristics described by the normal distribution model.

[0088] The overall probability value directly reflects the degree to which the distribution characteristics of the current real-time data match the distribution of historical normal data. When the business is operating normally and the data distribution has not shifted, the feature vector extracted from the real-time data should be similar to the features of the historical normal data during training, so the overall probability value should remain at a high level. When a distribution shift occurs and the data distribution characteristics change, the feature vector will deviate from the normal distribution model, and the overall probability value will drop significantly. This downward trend can serve as an important signal for detecting distribution shifts.

[0089] Finally, the calculated total probability value is used as the first offset determination index. The first offset determination index is a scalar value, taking the range of real numbers greater than 0, and typically remains at a relatively stable level under normal circumstances. This index characterizes the degree of deviation between the current data distribution and the normal distribution from the perspective of individual objects, and is one of the core indicators for implementing distribution offset detection.

[0090] The first offset criterion has a clear probabilistic meaning, representing the probability density of observing the current feature vector under the assumption of a normal distribution model. A higher criterion value indicates that the current data conforms more closely to the normal distribution pattern, and the lower the probability of offset. A lower criterion value indicates that the current data deviates more from the normal distribution pattern, and the higher the probability of offset. By setting a reasonable threshold, when the first offset criterion falls below the threshold, it can be determined that an individual-level distribution offset has occurred.

[0091] The first offset determination index is saved and passed to subsequent processing modules, including an adaptive threshold update module and a distribution offset determination module. In the adaptive threshold update module, this index is used to dynamically adjust the detection threshold using an exponentially weighted moving average algorithm. In the distribution offset determination module, this index is compared with the adaptive threshold as one of the criteria for determining whether a distribution offset has occurred. Furthermore, the time series data of the first offset determination index can also be used for subsequent trend prediction, helping to anticipate potential distribution offset risks in advance.

[0092] Example 5: In this embodiment, the real-time data distributions of the first and second detected objects are obtained. The real-time data distribution of the first detected object is represented as a first probability distribution, and the real-time data distribution of the second detected object is represented as a second probability distribution, thus obtaining the first and second probability distributions. Based on the first and second probability distributions, the minimum transmission cost to transform the first probability distribution into the second probability distribution is calculated, thus obtaining the Wasserstein distance. Based on the Wasserstein distance, the object similarity between the first and second detected objects is calculated using the formula S=1 / (1+W).

[0093] Specifically, the real-time data distribution of the first and second detected objects within the same group is first obtained. The same group refers to a collection of objects that have the same or similar attributes in business logic; theoretically, these objects should exhibit similar data distribution characteristics. For example, in e-commerce, different products under the same category constitute a group; in industrial settings, different equipment on the same production line constitutes a group; and in financial settings, different customers of the same product type constitute a group. Under normal circumstances, objects within the same group should have similar data distributions. When the distribution of some objects shifts, the consistency of the group decreases.

[0094] The system extracts real-time data for both the first and second detection objects within the current time window from the data source. This data can be raw business indicator values, such as product sales price sequences, equipment operating parameter sequences, or customer transaction amount sequences. The system then performs statistical processing on this raw data to generate a frequency distribution or probability distribution. For continuous data, the system can use kernel density estimation to estimate the probability density function. Kernel density estimation is a non-parametric method that places a kernel function (such as a Gaussian kernel) at each data point and then superimposes all kernel functions to obtain a smooth probability density curve. For discrete data or binned data, the frequency of each interval can be directly calculated and normalized to obtain the probability distribution.

[0095] The real-time data distribution of the first detected object is represented as a first probability distribution, and the real-time data distribution of the second detected object is represented as a second probability distribution. Probability distributions can be represented in various forms, including probability density functions, cumulative distribution functions, or discrete probability vectors. In practical implementations, continuous distributions are usually discretized, that is, the data space is divided into several intervals or grid points, and the probability value at each interval or grid point is calculated to form a discrete probability vector. This discretization not only captures the main characteristics of the distribution but also facilitates subsequent numerical calculations. The two probability distribution vectors have the same length, and the sum of their elements is equal to 1, respectively describing the distribution of the data of the two detected objects in each interval.

[0096] Secondly, based on the first and second probability distributions, the minimum transmission cost to transform the first probability distribution into the second probability distribution is calculated, yielding the Wasserstein distance. The Wasserstein distance is a method for measuring the difference between two probability distributions, also known as bulldozer distance or earth-moving distance. Intuitively, if we imagine the first probability distribution as the distribution of one pile of soil and the second probability distribution as the distribution of another pile of soil, the Wasserstein distance is the minimum amount of work required to redistribute the first pile of soil into the shape of the second pile; the amount of work equals the amount of soil moved multiplied by the distance moved.

[0097] Wasserstein distance calculation is based on optimal transmission theory, which studies how to transform one distribution into another with minimal cost. Mathematically, this is a linear programming problem, aiming to find the optimal transmission scheme, i.e., determining how much probability mass to transmit from each position in the first distribution to each position in the second distribution, such that the total transmission cost is minimized. The transmission cost consists of two parts: the probability mass of the transmission and the transmission distance; the sum of their products constitutes the total cost.

[0098] In the discretized representation, a distance matrix is ​​first constructed, describing the distance from the center point of each interval of the first probability distribution to the center point of each interval of the second probability distribution. The distance can be Euclidean distance, Manhattan distance, or other suitable metrics. Then, a linear programming problem is constructed, where the decision variable is the probability mass transferred from each interval of the first distribution to each interval of the second distribution. Constraints include that after the transfer, all probability masses from each interval of the first distribution are transferred out, the sum of the probability masses received by each interval of the second distribution equals its target probability value, and all transferred quantities are non-negative. The objective function is the sum of the products of all transferred probability masses and their corresponding distances. Minimizing this linear programming problem yields the Wasserstein distance.

[0099] The computation can be performed using a standard linear programming solver or a specialized Wasserstein distance calculation algorithm. For one-dimensional distributions, there are more efficient methods; the Wasserstein distance can be directly obtained by integrating the difference between the cumulative distribution functions of the two distributions. For multidimensional distributions, the computational complexity is higher, but several optimization algorithms are still available, such as the Sinkhorn algorithm, which accelerates the computation by introducing entropy regularization.

[0100] The calculated Wasserstein distance is a non-negative real number. A distance of 0 indicates that the two distributions are completely identical, while a larger distance indicates a greater difference between the two distributions. Compared to other distribution distance metrics (such as KL divergence or JS divergence), the Wasserstein distance has a significant advantage: it can measure the difference between non-overlapping distributions. Even if the support sets (regions with non-zero probabilities) of two distributions do not overlap at all, the Wasserstein distance can still provide a meaningful metric. This makes it particularly suitable for detecting distribution shifts, as shifts often lead to distribution displacements and non-overlapping conditions.

[0101] Finally, based on the calculated Wasserstein distance, a transformation formula is used to convert the distance value into a similarity value, yielding the object similarity between the first and second detected objects. The transformation formula is S = 1 / (1 + W), where S represents the similarity and W represents the Wasserstein distance. This transformation formula is a monotonically decreasing function that maps the distance space to the similarity space.

[0102] When the Wasserstein distance W is 0, it indicates that the two distributions are completely identical, and the similarity S equals 1, reaching its maximum value, meaning the two objects are completely similar. As the Wasserstein distance W increases, the similarity S decreases. As the distance approaches infinity, the similarity approaches 0, indicating that the two objects are completely dissimilar. This transformation maps unbounded distance values ​​to a bounded interval of 0 to 1, facilitating subsequent similarity network construction and threshold setting.

[0103] The choice of conversion formula reflects the inverse relationship between distance and similarity. The constant 1 in the denominator ensures that even with small distances, the similarity won't approach 1 and lose its discriminative power. In practical applications, the system can adjust the conversion formula according to business characteristics, for example, using variations such as S=exp(-W) or S=1 / (1+kW), where k is an adjustment parameter used to control the degree of influence of distance on similarity. However, the basic principle remains unchanged: the larger the distance, the smaller the similarity; and the smaller the distance, the larger the similarity.

[0104] The calculated object similarity reflects the degree of similarity in the data distribution of two detected objects within the same group within the current time window. Under normal circumstances, objects within the same group should have high similarity, indicating that they follow similar business patterns or operational rules. When some objects experience anomalies or distribution shifts, their similarity with other objects within the group decreases. This change in similarity provides important evidence for detecting distribution shifts at the group level. The similarity between each pair of objects within the group is calculated to form a similarity matrix, preparing for the subsequent construction of a similarity network.

[0105] Example 6: In this embodiment, each detected object within the same group is set as a node in the similarity network, resulting in a node set. Any two nodes in the node set are traversed, and when the similarity between the objects corresponding to the two nodes is greater than a preset similarity threshold, an edge is established between the two nodes, resulting in the similarity network. Based on the similarity network, a graph theory algorithm is used to identify connected clusters in the similarity network, and the proportion of the number of nodes in the largest connected cluster to the total number of nodes in the node set is calculated to obtain the connectivity. The connectivity is used as the second offset determination index to obtain the second offset determination index.

[0106] Specifically, each detection object within the same group is first set as a node in the similarity network, resulting in a node set. A similarity network is a graph structure used to represent the similarity relationships between objects; in graph theory, nodes represent entities, and edges represent relationships between entities. In the application scenario of this invention, nodes represent the business objects to be monitored, and edges represent similar data distribution characteristics between objects.

[0107] The size of a node set depends on the number of objects contained in the group. For example, in an e-commerce scenario, if monitoring is done on all products within a certain category, the node set would contain all products in that category, with each product corresponding to one node. In an industrial scenario, if monitoring is done on all equipment on a production line, the node set would contain all equipment on that production line, with each equipment corresponding to one node. The number of nodes typically ranges from tens to thousands, depending on the scale of the business and the granularity of the monitoring.

[0108] Each node is assigned a unique identifier to distinguish different objects and to record the object's attribute information, such as object name, object type, and object number. This information facilitates tracing the specific identity of abnormal objects during subsequent analysis. During the initialization phase of the graph structure, all nodes are isolated, with no edges connecting them, forming a completely disconnected graph. After the node set is constructed, the system prepares to establish edge relationships between nodes based on the similarity between objects.

[0109] Secondly, iterate through any two nodes in the node set. When the similarity between the objects corresponding to the two nodes is greater than a preset similarity threshold, establish an edge between the two nodes to obtain the complete similarity network. An edge indicates that the data distributions of the two objects have a sufficiently high similarity, and they can be considered to belong to the same distribution pattern or operating state.

[0110] The traversal process employs a double loop: the outer loop iterates through the first node, and the inner loop iterates through the second node, ensuring that every pair of nodes in the node set is checked once. For each pair of nodes, the system queries the similarity value between their corresponding objects from the similarity matrix. This similarity value is obtained through Wasserstein distance transformation in Example 5. The system compares the similarity value with a preset similarity threshold, which is a parameter pre-set by the system based on business characteristics and historical experience, typically set between 0.7 and 0.9, with a commonly used value of 0.8.

[0111] When the similarity value is greater than a preset similarity threshold, it indicates that the distributions of the two objects are sufficiently similar, and the system establishes an undirected edge between the corresponding two nodes. An undirected edge signifies that the similarity relationship is symmetrical; that is, if object A is similar to object B, then object B is also similar to object A. Establishing the edge connects the two nodes in the graph structure, forming a connected relationship. When the similarity value is less than or equal to the preset similarity threshold, the system does not establish an edge, and the two nodes remain disconnected in the graph.

[0112] After traversing all node pairs, a complete similarity network is obtained. This network is an undirected graph, where nodes represent objects to be detected and edges represent high similarity relationships between objects. Under normal circumstances, because objects within a group follow similar business patterns, the similarity between most objects will exceed a threshold, resulting in densely connected edges in the network, with most nodes interconnected, forming one or a few large connected clusters. When a distribution shift occurs, the similarity between the shifted object and the normal object decreases, causing the corresponding edges to disappear, reducing the network's connectivity, and potentially leading to phenomena such as connected cluster splitting and an increase in isolated nodes. The changes in the topology of the similarity network directly reflect changes in the consistency of group distribution, providing a graph structure perspective for detecting anomalies at the group level.

[0113] Next, based on the constructed similarity network, graph theory algorithms are used to identify connected clusters in the network. The connectivity is obtained by calculating the proportion of nodes in the largest connected cluster to the total number of nodes. A connected cluster is a concept in graph theory, referring to a subset of nodes in a graph where any two nodes in the subset can be reached by a path, and the subset is not connected to any other nodes in the graph. A graph can contain one or more connected clusters, and isolated nodes (nodes without any edges) also constitute a single-node connected cluster.

[0114] Graph traversal algorithms such as depth-first search or breadth-first search are used to identify connected clusters. The basic algorithm proceeds as follows: starting from an unvisited node, all nodes connected to it are visited via edges, these nodes are marked as visited and added to the same connected cluster. Then, the next unvisited node is selected, and the above process is repeated until all nodes have been visited. Each time a new unvisited node is started, a new connected cluster is discovered. After traversal is complete, a list of all connected clusters is obtained, and each connected cluster records the set of nodes it contains.

[0115] Analyzing all connected clusters, the cluster containing the largest number of nodes is identified, called the maximal connected cluster. The maximal connected cluster represents the largest connected group in the network; under normal circumstances, this maximal connected cluster should contain the vast majority of objects in the group. The system calculates the number of nodes contained in the maximal connected cluster and then divides it by the total number of nodes in the cluster to obtain the connectivity. Connectivity is a ratio between 0 and 1; the closer the value is to 1, the better the network connectivity and the higher the consistency of the group. The closer the value is to 0, the worse the network connectivity and the lower the consistency of the group.

[0116] For example, if a group contains 100 objects, under normal circumstances, 95 objects have similar data distributions, forming a large connected cluster in the similarity network, with a connectivity of 0.95. If a group distribution shift occurs, only 60 objects maintain the normal distribution, while the distribution of the remaining 40 objects shifts. The network may split into a maximum connected cluster of 60 nodes and several smaller connected clusters or isolated nodes, at which point the connectivity drops to 0.60. A significant decrease in connectivity indicates reduced group consistency, potentially suggesting systemic risks or batch anomalies.

[0117] Finally, the calculated connectivity is used as the second offset metric. This second offset metric characterizes the consistency of data distribution at the group level, complementing the first offset metric. The first offset metric focuses on whether the distribution of an individual object deviates from the normal model, while the second offset metric focuses on whether the distribution of objects within a group remains consistent. Using both metrics together allows for a more comprehensive detection of distribution offsets.

[0118] The second offset metric has clear graph theory and business implications. From a graph theory perspective, it measures the connectivity strength of a network, reflecting the integrity of a network of similar relationships. From a business perspective, it reflects the degree to which objects within a group follow a unified pattern, serving as a quantitative indicator of group behavioral consistency. Under normal circumstances, the second offset metric should remain at a high level, typically above 0.8. When group anomalies occur, such as system failures causing multiple devices to malfunction simultaneously, market changes leading to multiple products becoming unsaleable simultaneously, or policy adjustments causing multiple customers to be lost simultaneously, the second offset metric will decrease significantly.

[0119] The second offset determination index is saved and passed to subsequent processing modules. In the adaptive threshold update module, this index is used to dynamically adjust the detection threshold using an exponentially weighted moving average algorithm, ensuring that the threshold can adapt to normal fluctuations in group behavior. In the distribution offset determination module, this index is compared with the adaptive threshold. When the index value is lower than the threshold, a distribution offset at the group level is determined, triggering an alarm mechanism. Furthermore, the time series data of the second offset determination index can also be used for trend analysis and prediction, helping the system to identify declining trends in group consistency in advance and providing decision support for preventative intervention. By introducing the second offset determination index, the gap in traditional techniques that ignore group relationships is filled, achieving an organic combination of individual detection and group detection, significantly improving the comprehensiveness and accuracy of distribution offset detection.

[0120] Example 7: Obtain the first offset determination index at the current time and the first adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the first adaptive threshold at the current time; obtain the second offset determination index at the current time and the second adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the second adaptive threshold at the current time.

[0121] Specifically, the first offset judgment index at the current moment and the first adaptive threshold at the previous moment are obtained first. The smoothing coefficient α is set to 0.3, and the first adaptive threshold at the current moment is calculated. The exponentially weighted moving average algorithm is a time series smoothing technique widely used in data analysis and forecasting. Its core idea is to assign exponentially decaying weights to historical data, with more recent data having larger weights and more distant data having smaller weights, thereby achieving adaptive tracking of the time series.

[0122] The smoothing coefficient α is a key parameter in the exponentially weighted moving average algorithm, ranging from 0 to 1. It determines the weighting of the current value and historical values ​​in the new average. A larger α value results in a greater weight for the current value, leading to a faster response to data changes, but also greater susceptibility to noise. A smaller α value results in a greater weight for historical values, stronger smoothing effect, but a slower response to data changes. This invention sets α to 0.3, a compromise proven in practice, which allows for rapid response to real business changes while effectively filtering out short-term fluctuations and noise.

[0123] First, the system obtains the first offset determination index for the current moment. This index, obtained in Example 4, is calculated by determining the total probability value of the feature vector belonging to the normal distribution model, reflecting the degree of matching between the distribution characteristics of the current real-time data and the distribution of historical normal data. Then, the system obtains the first adaptive threshold from the previous moment. This threshold is calculated during the last update and stored in the system's state variables. When the system first runs, there is no threshold from the previous moment; the system uses the statistical value (such as the mean or lower quantile) of the first offset determination index of historical normal data as the initial threshold.

[0124] The first offset criterion at the current time is multiplied by a smoothing coefficient of 0.3 to obtain the weighted contribution of the current value. Simultaneously, the first adaptive threshold at the previous time is multiplied by 1 and subtracted from the smoothing coefficient (0.7) to obtain the weighted contribution of historical values. The two weighted contributions are then added together to obtain the first adaptive threshold at the current time. This calculation process reflects the essence of the exponentially weighted moving average, where the new threshold is a weighted combination of the current observation and the historical average, with a weight ratio of 3:7.

[0125] This calculation method allows the first adaptive threshold to dynamically adjust as the first offset determination metric changes. During normal business fluctuations, such as daily promotions on e-commerce platforms causing changes in data distribution but still within the normal range, the first offset determination metric changes accordingly, and the adaptive threshold adjusts accordingly to avoid misjudging normal fluctuations as abnormal offsets. When a genuine distribution offset occurs, the first offset determination metric will remain significantly below the threshold; even if the threshold decreases slowly, the offset signal can still be detected. This adaptive mechanism solves the problem of traditional fixed thresholds being unable to adapt to business changes, significantly reducing false positive and false negative rates and minimizing the workload of manually adjusting thresholds.

[0126] Next, the second offset determination index at the current time and the second adaptive threshold at the previous time are obtained. The smoothing coefficient α is set to 0.3, and the second adaptive threshold at the current time is calculated. The calculation method of the second adaptive threshold is exactly the same as that of the first adaptive threshold, both using the exponentially weighted moving average algorithm and the same smoothing coefficient of 0.3 to ensure that the update logic of the two thresholds is consistent.

[0127] First, the second offset criterion for the current time step is obtained. This criterion, obtained in Example 6 by calculating the connectivity of the similarity network, reflects the consistency of the distribution of object data within the group. Then, the second adaptive threshold for the previous time step is obtained; this threshold was calculated during the last update. During the initial system run, the statistical value of the second offset criterion for historical normal data is used as the initial threshold. Typically, the mean of historical connectivity minus a standard deviation, or a lower quantile of historical connectivity, can be selected to ensure the initial threshold is set at a reasonable level.

[0128] The same calculation process as the first adaptive threshold is performed: the second offset determination index at the current time is multiplied by 0.3, and the second adaptive threshold at the previous time is multiplied by 0.7. These two values ​​are then added together to obtain the second adaptive threshold at the current time. This new threshold comprehensively considers both the actual level of current group connectivity and the trend of historical connectivity, achieving adaptive tracking of changes in group behavior.

[0129] The second adaptive threshold's dynamic update mechanism enables the system to adapt to the reasonable evolution of group behavior patterns. For example, in industrial production scenarios, as equipment ages or processes are adjusted, the distribution of operating parameters of the equipment group may drift slowly overall. However, as long as the group remains consistent, alarms should not be triggered. The adaptive threshold can adjust to follow this slow drift, avoiding false alarms. Simultaneously, if a sudden failure causes the operating parameters of some equipment to deviate sharply from the group, connectivity will rapidly decrease and remain below the threshold. Even as the threshold gradually decreases, the offset signal will still be captured promptly, ensuring detection sensitivity.

[0130] The design of two adaptive thresholds fully reflects the intelligent features of this invention. The system does not require manual preset of fixed thresholds or frequent manual adjustments. Instead, it automatically learns the normal fluctuation pattern of data through algorithms and adaptively adjusts the judgment criteria when business changes. While maintaining high detection accuracy, it significantly reduces operation and maintenance costs, and truly realizes the automation and intelligence of distribution offset detection.

[0131] Example 8: In this embodiment, the duration of the sliding window is 24 hours.

[0132] Specifically, the sliding window duration is set to 24 hours, a decision based on a combination of the periodicity of business operations and the real-time requirements of monitoring. A sliding window is a crucial concept in time series analysis, referring to a fixed-length time interval that continuously moves forward, always covering the most recent period of data. The introduction of a sliding window allows the system to focus on recent data, promptly reflecting the current business status and avoiding interference from excessively distant historical data.

[0133] The 24-hour window length setting fully considers the daily cycle characteristics of most business scenarios. Many business activities exhibit obvious daily cycle patterns. For example, e-commerce platform transaction volume peaks during the day and troughs at night; industrial equipment load is high on weekdays and low on holidays; and financial transactions are active during trading hours and stagnant outside of trading hours. A complete 24-hour cycle can cover all typical business states within a day, including peak, off-peak, and low-peak periods, ensuring that the statistical characteristics and thresholds calculated by the system reflect the complete business cycle and avoiding detection bias caused by sampling time period deviations.

[0134] A 24-hour window length strikes a good balance between data volume and response speed. A longer window includes more data points and provides more stable estimates of statistical characteristics, but it responds more slowly to changes in distribution. A shorter window responds faster to changes in distribution, but statistical characteristics are more susceptible to random fluctuations and have lower stability. The 24-hour setting ensures sufficient data volume to support reliable statistical analysis (typically including thousands to tens of thousands of data points) while also ensuring timely response to distribution shifts, detecting abnormal changes and triggering alerts within a day, thus meeting the real-time requirements of most business scenarios.

[0135] In practical applications, the sliding window is implemented by maintaining a data queue indexed by timestamps. Whenever new data arrives, it is added to the tail of the queue, while the timestamp of the data at the head of the queue is checked. If it exceeds 24 hours, it is removed from the queue. This ensures the queue always contains data from the most recent 24 hours, and the queue continuously slides and updates over time. Based on the data within this sliding window, adaptive thresholds are calculated, normal data is filtered for incremental model updates, and time-series features are extracted for trend prediction.

[0136] Example 9: In this embodiment, the first offset determination index and the first adaptive threshold are obtained, and it is determined whether the first offset determination index is lower than the first adaptive threshold to obtain a first determination result; the second offset determination index and the second adaptive threshold are obtained, and it is determined whether the second offset determination index is lower than the second adaptive threshold to obtain a second determination result; when the first determination result is true or the second determination result is true, the current detection object is marked as having a distribution offset, and the distribution offset alarm is generated.

[0137] Specifically, the first step is to obtain a first offset determination index and a first adaptive threshold. Then, it is determined whether the first offset determination index is lower than the first adaptive threshold, thus obtaining a first determination result. This is the core step in detecting the distribution offset of individual objects. By comparing the current offset determination index with the dynamically adjusted threshold, it is determined whether the data distribution of individual objects has deviated abnormally.

[0138] The first offset determination index for the current moment is obtained from the state management module. This index, calculated in Example 4, represents the total probability value that the current feature vector belongs to the normal distribution model. Simultaneously, the first adaptive threshold, which was just updated in Example 7, is obtained. This threshold is dynamically adjusted using an exponentially weighted moving average algorithm, reflecting the normal level of the first offset determination index in recent times.

[0139] A numerical comparison operation is performed to determine whether the first offset judgment index is strictly less than the first adaptive threshold. If the first offset judgment index is lower than the threshold, it indicates that the matching degree between the current data distribution characteristics and the normal distribution model has significantly decreased, falling below the normal level, and there are signs of distribution offset. In this case, the first judgment result is true. If the first offset judgment index is greater than or equal to the threshold, it indicates that the current data distribution characteristics are still within the normal range and there is no obvious offset. In this case, the first judgment result is false.

[0140] The initial determination of truth typically corresponds to the following business scenarios: a substantial change in the data distribution of an individual object, such as the price distribution of a product changing from a normal distribution to a bimodal distribution, the operating parameters of a device changing from a stable state to a state of violent fluctuation, or a customer's transaction behavior changing from regularity to randomness. This change causes the feature vector extracted from real-time data to deviate from the feature space described by the normal distribution model, resulting in a decrease in the overall probability value. The introduction of an adaptive threshold ensures the accuracy of the determination. Even when normal business fluctuations cause changes in the overall probability value, the threshold will be adjusted accordingly. Only when the decrease in the probability value exceeds the normal fluctuation range will the initial determination of truth be triggered.

[0141] Next, the second offset determination index and the second adaptive threshold are obtained. It is then determined whether the second offset determination index is lower than the second adaptive threshold to obtain the second determination result. This is the core step in detecting distribution offset at the group level. By comparing the group connectivity with the dynamic threshold, it is determined whether the distribution consistency of objects within the group has been abnormally reduced.

[0142] The system obtains the second offset determination index for the current moment from the state management module. This index, obtained in Example 6 by calculating the connectivity of similar networks, represents the proportion of the largest connected cluster in the group to the total number of objects. Simultaneously, the system obtains the second adaptive threshold updated in Example 7, which reflects the normal level of recent group connectivity.

[0143] A numerical comparison operation is performed to determine whether the second offset criterion is strictly less than the second adaptive threshold. If the second offset criterion is lower than the threshold, it indicates a significant decrease in the connectivity of the similarity network, a reduction in the consistency of object distribution within the group, and signs of group anomalies; in this case, the second judgment result is true. If the second offset criterion is greater than or equal to the threshold, it indicates that the group connectivity is still within the normal range, and the group consistency remains good; in this case, the second judgment result is false.

[0144] The second judgment result being true typically corresponds to the following business scenarios: a systematic shift in the data distribution of some objects within a group leads to a split in the similarity network. For example, in e-commerce, some products in a certain category may simultaneously be out of stock due to supply chain issues, resulting in abnormal sales data distribution. These products lose connection with normal products in the similarity network, forming isolated nodes or small connected clusters, reducing the size of the largest connected cluster and decreasing connectivity. In industrial scenarios, some equipment on a production line may simultaneously experience shifts in operating parameters due to common environmental factors (such as increased temperature or changes in raw material batches). The similarity between these devices and normal equipment decreases, leading to reduced network connectivity. The second judgment result can effectively identify such group anomalies, compensating for the shortcomings of the first judgment result, which only focuses on individuals.

[0145] Finally, a logical OR operation is used to combine the two judgment results. When either the first judgment result or the second judgment result is true, a distribution shift is marked for the currently detected object, and a distribution shift alarm is generated. This comprehensive judgment mechanism achieves an organic integration of individual detection and group detection, capturing anomalies at both the individual and group levels, significantly improving the comprehensiveness of detection.

[0146] The logical OR operation means that if at least one of the two judgment results is true, the overall judgment result is true, indicating a distribution shift has occurred. This design is based on the following considerations: individual and group shifts can occur individually or simultaneously, requiring a response to all situations. When only the first judgment result is true, it indicates a shift in the distribution of individual objects, but the group as a whole remains consistent. This may be a special case for individual objects, requiring attention but not necessarily a systemic risk. When only the second judgment result is true, it indicates a decrease in group connectivity, but this may be due to the accumulation of small shifts from multiple objects, and the first shift judgment indicator for a single object may not have yet triggered the threshold. When both judgment results are true, it indicates both a significant shift at the individual level and a decrease in consistency at the group level. This is the most severe anomaly, potentially indicating a systemic failure or significant business changes.

[0147] When the overall judgment result is true, the alarm generation process is executed. First, the current detected object is marked as having a distribution offset, and an offset tag and offset detection timestamp are added to the object's status record. Then, the system generates a distribution offset alarm message, which contains key information such as alarm time, alarm object identifier, alarm type (individual offset, group offset, or comprehensive offset), the current value and threshold of the first offset judgment indicator, the current value and threshold of the second offset judgment indicator, and the severity assessment of the offset.

[0148] Alarm messages are pushed to relevant personnel or systems through multiple channels. These channels can include real-time alarm displays on a visual monitoring interface, email notifications, SMS notifications, instant messaging push notifications, and API calls to downstream business systems. Alarm message sending follows deduplication and aggregation principles to avoid information overload caused by repeatedly sending a large number of identical alarms within a short period. For example, the system can set an alarm silence period, where no duplicate alarms are sent for a certain period after an object triggers an alarm (e.g., 10 minutes), unless the offset worsens further. For alarms triggered simultaneously by multiple objects, the system can aggregate them to generate batch alarm messages, indicating how many objects have experienced distribution offsets and providing an overall overview.

[0149] Once an alarm is generated, subsequent root cause analysis and trend prediction processes will be initiated to provide users with more in-depth analytical support, helping to quickly locate the cause of the problem and assess future risks, supporting business decisions and emergency response. The entire alarm mechanism is designed with practicality and operability in mind, ensuring that detected anomalies can be promptly, accurately, and effectively communicated to those who need to know, truly realizing the value of distribution offset detection in business assurance.

[0150] Example 10: In this embodiment, data with the first offset criterion higher than the first adaptive threshold and the second offset criterion higher than the second adaptive threshold are filtered from the sliding window to obtain incremental training data; based on the incremental training data and the parameters of the current Gaussian mixture model, an incremental expectation step is performed to calculate the posterior probability of each data point in the incremental training data belonging to each sub-distribution, resulting in an incremental membership matrix; based on the incremental membership matrix and the incremental training data, an incremental maximization step is performed to update the weights, mean vectors, and covariance matrices of each sub-distribution, resulting in candidate update parameters; according to the candidate update parameters, the rate of change of the mean vector in the candidate update parameters relative to the mean vector in the parameters of the current Gaussian mixture model is calculated, and it is determined whether the rate of change is less than or equal to 10%, resulting in a stability determination result; when the stability determination result is true, the parameters of the current Gaussian mixture model are replaced with the candidate update parameters to obtain the updated normal distribution model.

[0151] Specifically, data with a first offset metric higher than a first adaptive threshold and a second offset metric higher than a second adaptive threshold are first filtered from a sliding window to obtain incremental training data. This filtering process ensures that the data used for model updates are normal data, avoiding the inclusion of abnormal data in training and thus preventing model contamination.

[0152] The sliding window contains all real-time data from the past 24 hours, along with their corresponding offset criteria and thresholds. The system iterates through each data record in the window, checking whether the first offset criterion for that data is higher than the current first adaptive threshold, and simultaneously checking whether the second offset criterion is higher than the current second adaptive threshold. Only when both conditions are met is the data considered normal and selected for the incremental training dataset.

[0153] This dual-screening mechanism provides rigorous data quality assurance. A first offset criterion exceeding a threshold indicates that the data point conforms to a normal distribution model at the individual level, with no individual anomalies. A second offset criterion exceeding a threshold indicates that the population connectivity at the corresponding time point is normal, with no population anomalies. The simultaneous fulfillment of both conditions ensures that the selected data is normal at both the individual and population levels, representing high-quality training samples.

[0154] The incremental training dataset obtained through filtering typically contains most of the data in the sliding window, potentially reaching 80% to 90% or more during normal operation. During periods of distribution shift, the filtering ratio decreases as outliers are removed. The system records the amount of incremental training data. If the amount is too small (e.g., less than 20% of the total window data), it indicates a potential persistent and severe anomaly. In this case, the system pauses model updates to prevent forced updates that could lead to model failure due to an excessively high proportion of outlier data.

[0155] The selected incremental training data will be used in the subsequent incremental learning process. This data represents the recent normal business status. Through incremental learning, the model parameters can gradually adapt to the slow evolution of the business, such as changes in user habits, changes in the market environment, and concept drift phenomena such as aging of equipment performance, thus maintaining the long-term effectiveness of the model.

[0156] Secondly, based on the incremental training data and the current Gaussian mixture model parameters, an incremental expectation step is performed to calculate the posterior probability of each data point in the incremental training data belonging to each sub-distribution, thus obtaining the incremental membership matrix. The incremental expectation step is the same in principle as the standard expectation step in Example 3, the difference being that the datasets processed are different. The standard expectation step processes complete historical normal data, while the incremental expectation step processes newly added incremental training data.

[0157] The system obtains the parameters of the current Gaussian mixture model, including the weights, mean vector, and covariance matrix of each sub-distribution. These parameters are obtained from the previous training or update and represent the model's current understanding of the normal distribution. For each data point in the incremental training data, the system calculates its probability density value under each sub-distribution, then multiplies it by the weight of the corresponding sub-distribution to obtain the joint probability. Dividing the joint probability corresponding to a certain sub-distribution by the sum of the joint probabilities of all sub-distributions yields the posterior probability that the data point belongs to that sub-distribution.

[0158] Repeat the above calculations to compute the posterior probabilities for all data points and all subdistributions in the incremental training data, forming the incremental membership matrix. The number of rows in this matrix equals the number of samples in the incremental training data, the number of columns equals the number of subdistributions, and the matrix elements represent the degree to which each data point belongs to each subdistribution. The incremental membership matrix provides the soft assignment relationship between data points and subdistributions, providing a basis for subsequent parameter updates.

[0159] The computational complexity of the incremental expectation step is proportional to the size of the incremental training data. Since the amount of incremental data is much smaller than the total historical data, the computational overhead of the incremental expectation step is relatively small, enabling a fast response. This is a significant advantage of incremental learning over full retraining. Incremental learning only needs to process the new data, avoiding the computational burden of repeatedly processing massive amounts of historical data, thus significantly improving the system's real-time performance and efficiency.

[0160] Next, based on the incremental membership matrix and incremental training data, an incremental maximization step is performed to update the weights, mean vectors, and covariance matrices of each sub-distribution, obtaining candidate update parameters. The incremental maximization step is in principle the same as the standard maximization step in Example 3, but in practice, an incremental update strategy is adopted to integrate the information of the new data into the existing parameters, rather than completely recalculating the parameters.

[0161] For incremental updates of sub-distribution weights, the sum of the membership degrees of the incremental training data under each sub-distribution is calculated, and then weighted and averaged with the existing weights to obtain the new weights. The weight coefficients of the weighted average can be based on the proportion of data volume, for example, determining the influence weight of the new data based on the proportion of the incremental data volume to the sum of the historical data volume and the incremental data volume. This incremental update method considers both the contribution of historical data to the weights and incorporates information from the new data, achieving a smooth transition.

[0162] For incremental updates to the mean vector, the weighted mean of the incremental training data under each sub-distribution is first calculated. This is done by summing the products of each data point and its membership degree, and then dividing by the sum of the membership degrees. This incremental mean is then weighted and averaged with the existing mean vector to obtain the new mean vector. The weighting coefficients are also based on the proportion of the data volume to ensure that the new mean comprehensively reflects the distribution center of both historical and incremental data.

[0163] For incremental updates of the covariance matrix, the system first calculates the covariance matrix of the incremental training data under each sub-distribution, which is calculated by weighting the membership degrees of the difference vector between the data point and the incremental mean and its outer product. Then, the incremental covariance matrix is ​​weighted and averaged with the existing covariance matrix to obtain the new covariance matrix. This process ensures that the new covariance matrix retains both the dispersion characteristics of historical data and incorporates the dispersion information of incremental data.

[0164] After the incremental maximization step is completed, candidate update parameters are obtained, including candidate weights, candidate mean vectors, and candidate covariance matrices. Compared with the current parameters, these candidate parameters incorporate the distribution characteristics of recent normal data and can better reflect the current business status. However, before directly adopting the candidate parameters, a stability test is required to ensure that the parameter updates are not too drastic and cause model failure.

[0165] Finally, based on the candidate update parameters, the rate of change of the mean vector in the candidate update parameters relative to the mean vector in the current Gaussian mixture model parameters is calculated, and it is determined whether the rate of change is less than or equal to 10%, thus obtaining the stability judgment result. This stability check mechanism is an important protective measure for incremental learning, preventing the model parameters from changing too much in a short period of time, which could lead to a sharp decline in model performance or failure.

[0166] First, extract the mean vectors of all sub-distributions in the candidate update parameters, as well as the mean vectors of all sub-distributions in the current model parameters. For each sub-distribution, calculate the difference between the candidate mean vector and the current mean vector, and then calculate the norm of the difference (e.g., Euclidean distance) as the absolute change. Divide the absolute change by the norm of the current mean vector to obtain the relative rate of change, which is a dimensionless proportional value reflecting the relative magnitude of the change in the mean vector.

[0167] Calculate the relative rate of change for all sub-distributions, and then take the maximum value as the overall rate of change. Compare the overall rate of change with a preset stability threshold of 10%. If the rate of change is less than or equal to 10%, it indicates that the parameter update magnitude is moderate, the model can evolve smoothly, and the stability judgment result is true. If the rate of change is greater than 10%, it indicates that the parameter update magnitude is too large, which may be due to a large difference between the incremental data and historical data, or insufficient incremental data leading to statistical instability, and the stability judgment result is false.

[0168] A 10% stability threshold is a reasonable value that has been proven in practice. It allows for a certain degree of adaptive adjustment of model parameters, enabling the tracking of slow business drift while preventing model runaway caused by sudden parameter mutations. In different application scenarios, this threshold can be adjusted according to business characteristics. For rapidly changing businesses, the threshold can be appropriately increased, while for scenarios with extremely high stability requirements, the threshold can be appropriately decreased.

[0169] Stability assessment not only focuses on changes in the mean vector, but in practice, it can also be extended to check changes in the weights and covariance matrix to ensure the stability of the overall parameters. For example, it can check whether the changes in weights exceed a certain proportion, or whether the changes in the determinant of the covariance matrix exceed a certain multiple, providing multi-dimensional stability assurance.

[0170] Finally, when the stability determination result is true, the current Gaussian mixture model parameters are replaced with candidate update parameters to obtain the updated normal distribution model. This is the final step in the incremental learning process, completing the actual model update and ensuring that the model accurately describes the current business state.

[0171] All sub-distribution weights, mean vectors, and covariance matrices in the candidate update parameters are copied completely to the parameter storage area of ​​the normal distribution model, replacing the original parameter values. After the replacement, the normal distribution model contains the latest parameters, which combine the distribution characteristics of historical training data and recent normal data, representing the model's latest understanding of the current normal distribution.

[0172] The updated normal distribution model will be immediately applied to subsequent real-time data detection. Newly arrived real-time data will be used to calculate the first offset judgment index based on the new model, ensuring that detection is always based on the latest normal benchmark. This continuous update mechanism enables the model to dynamically adapt to business evolution, effectively address the concept drift problem, and maintain long-term detection accuracy.

[0173] Record metadata such as the timestamp of model updates, the amount of incremental data used in the updates, and the rate of parameter change for model management and performance monitoring. By analyzing the frequency of model updates and the trend of parameter changes, we can understand the stability and evolution speed of the business, providing a reference for system optimization.

[0174] When the stability assessment result is false, the parameter update is rejected, and the current model parameters remain unchanged. The event and reason for the update rejection will be recorded and monitored as an anomaly signal. Multiple consecutive update rejections may indicate that the business is undergoing rapid changes or that there is a persistent anomaly, which may trigger special handling procedures, such as sending a notification to the administrator suggesting manual intervention and evaluation, or performing a full retraining using accumulated normal data after the anomaly ends to restore the model's effectiveness.

[0175] The complete process of the incremental learning mechanism reflects the intelligent design of this invention in terms of model maintenance. The system does not require manual retraining of the model periodically, but automatically, continuously and smoothly updates the model parameters, which not only ensures the timeliness of the model, but also ensures its stability. While greatly reducing the operation and maintenance costs, it ensures the long-term reliability of the detection performance and truly realizes the adaptive and maintenance-free operation of the distributed migration detection system.

[0176] Example 11: In this embodiment, business logic rules and historical offset event data are acquired, and a directed acyclic graph is constructed based on the business logic rules and historical offset event data to obtain a causal graph. The result nodes of the causal graph are the first offset judgment index and the second offset judgment index, and the influencing factor nodes are business variables. When the distributed offset alarm is output, all influencing factor nodes are obtained from the causal graph, and an intervention operation is performed on each influencing factor node to fix its value. The change magnitude of the first offset judgment index and the second offset judgment index in the causal graph after the intervention is calculated. Based on the change magnitude, all influencing factor nodes are sorted in descending order, and the top three influencing factor nodes with the largest change magnitudes are selected to obtain the key root causes. The key root causes are output to a visualization interface to obtain the root cause analysis results.

[0177] Specifically, the first step is to acquire business logic rules and historical offset event data to provide a knowledge foundation and data support for building a business-driven causal graph. Business logic rules are rule-based knowledge summarized by domain experts based on their business understanding. They describe the causal relationships and influence mechanisms between business variables. These rules represent the experience and insights of human experts and are an important source of prior knowledge for building causal graphs.

[0178] The acquisition of business logic rules involves several approaches. First, rules are extracted from business documents, including business process descriptions, operation manuals, and system design documents. These documents typically contain explicit descriptions of business logic. For example, in an e-commerce scenario, the document might state that "product prices affect sales volume, insufficient inventory leads to increased stockout rates, and promotional activities increase website traffic." These descriptions can be transformed into causal rules. Then, natural language processing techniques, such as dependency parsing and relation extraction, are used to automatically identify causal relationships in the documents, extract subject-verb-object triples, and construct a preliminary rule set.

[0179] Furthermore, business logic rules were obtained through expert interviews. Business domain experts, system architects, and senior operations and maintenance personnel were invited to participate in knowledge collection. Through structured interviews or questionnaires, they were asked which factors they believed would affect the system's key indicators, whether the impact was positive or negative, the strength of the impact, and whether it was a direct or indirect impact. The knowledge provided by the experts was organized into rule form, such as "When raw material prices rise, production costs increase," and "When equipment temperature exceeds a threshold, equipment failure rate increases," thus establishing a rule knowledge base.

[0180] Historical offset event data refers to a complete record of all distributed offset events detected by the system in the past, including the offset occurrence time, offset object, offset type, values ​​of various business variables at the time of offset, values ​​of the first and second offset judgment indicators, manually confirmed root causes, corrective measures taken, and recovery status. This historical data is a valuable asset of experience, containing empirical evidence of causal relationships in real business environments.

[0181] All offset event records are extracted from the historical event database. For each event, the system obtains time-series data of all business variables within a time window before and after the event's occurrence. Business variables include raw business metrics such as product price, sales volume, inventory, and ratings; equipment temperature, pressure, speed, and energy consumption; user traffic, purchase volume, activity level, and retention rate; as well as external environmental variables such as weather conditions, holiday markers, market indices, and competitor activities. Historical values ​​of the first and second offset judgment indicators corresponding to the event are also extracted. These indicators serve as the outcome variables in the causal graph and are the targets of causal analysis.

[0182] Historical offset events are annotated and enhanced. If the event record contains manually confirmed root cause information, the root cause is annotated to the corresponding business variable, establishing a root cause-event mapping relationship. If the historical record lacks root cause annotations, the system can use the causal discovery method in Example 11 to perform backtracking analysis on historical data to infer possible root causes, or invite experts to supplement the annotations of historical events to improve data quality. By accumulating a large number of annotated historical events, causal patterns can be learned from the data, and it can be discovered which abnormal changes in business variables frequently lead to distribution offsets. These statistical regularities corroborate and complement the expert rules.

[0183] Secondly, a directed acyclic graph (DAG) is constructed based on business logic rules and historical offset event data to obtain a causal graph. The result nodes in the causal graph serve as the first and second offset judgment indicators, while the influencing factor nodes represent business variables. A DAG is a graph structure composed of nodes and directed edges. Nodes represent variables, and directed edges represent causal relationships, pointing from the cause variable to the result variable. There are no cycles in the graph; that is, starting from any node and following a directed edge, it is impossible to return to the original node. This ensures the unidirectionality and logical consistency of causal relationships.

[0184] The set of nodes in the cause-effect graph is determined, and the nodes are divided into two categories: result nodes and influencing factor nodes. Result nodes are the target variables for causal analysis, and in this invention, they are set as the first offset criterion and the second offset criterion. These two criteria are direct measures of distribution offset, and the purpose of the analysis is to find the root causes affecting these two criteria. Influencing factor nodes are business variables that may affect result nodes. All acquired business variables are set as potential influencing factor nodes, forming a complete set of nodes.

[0185] The system defines the set of edges in the causal graph, where edges represent causal relationships between nodes. A hybrid approach combining business logic rules and data-driven causal discovery is used to construct these edges. For causal relationships explicitly described in business logic rules, the system directly adds directed edges between the corresponding nodes. For example, if the rule states "price affects sales volume," a directed edge from price to sales volume is established between the price node and the sales volume node. These rule-based edges represent prior knowledge and reflect a deterministic causal mechanism.

[0186] For variable relationships not covered by the rules, causal discovery is performed using historical offset event data. The system employs constraint-based causal discovery algorithms, such as the PC algorithm or FCI algorithm, to learn the causal structure from the data. The basic principle of the PC algorithm is to identify causal relationships through conditional independence tests. If variables A and B are not independent given any other set of variables, then there is a direct causal relationship or a common cause relationship between A and B, and the causal direction is determined through further orientation rules. The PC algorithm is executed on all variables in the historical event data to obtain a data-driven causal graph skeleton.

[0187] Rule-based edges and data-based edges are merged to construct a comprehensive causal graph. During the fusion process, if the rules and data agree on the causal relationship of the same pair of variables, the system directly adopts that edge and increases its confidence level. If the rules and data disagree—for example, the rule states that A influences B but the data does not show this relationship, or the data shows that A influences B but the rule does not describe it—the system marks these conflicts. These conflicts can be resolved through expert review, or both possibilities can be retained and verified in subsequent analyses.

[0188] It is also necessary to ensure that the constructed graph is a directed acyclic graph, i.e., free of cycles. The existence of a cycle implies a closed causal relationship, such as A influencing B, B influencing C, and C influencing A. This is logically contradictory, as causal relationships should have a temporal order. A topological sorting algorithm is used to detect cycles in the graph. If a cycle is found, the edges within the cycle are analyzed. Combining timestamp information or expert knowledge, the least reliable edges are removed to break the cycle, ensuring the graph's acyclicity.

[0189] In a cause-and-effect diagram, influencing factor nodes may exhibit hierarchical relationships. Some influencing factors directly affect the outcome node, known as direct causes or proximate causes, while others indirectly affect the outcome node by influencing other influencing factors, known as indirect causes or distant causes. For example, in an industrial scenario, external temperature is a distant cause, affecting the intermediate variable of workshop temperature. Workshop temperature then affects equipment temperature, which ultimately affects the first offset judgment index. Cause-and-effect diagrams fully express this hierarchical causal propagation path, providing a structured analytical framework for subsequent root cause localization.

[0190] The completed causal graph is stored as a graph data structure. Nodes record attributes such as variable name, variable type, and current value, while edges record attributes such as causal strength, confidence level, and source (rule or data). Graph visualization is provided, using graph layout algorithms to present the causal graph in a hierarchical or networked manner. Result nodes are typically placed on the far right or at the bottom of the graph, while influencing factor nodes are arranged from left to right or top to bottom according to hierarchical relationships. Arrows indicate the causal direction, facilitating users' intuitive understanding of the causal structure of the business system.

[0191] Next, when an output distribution offset alarm is triggered, all influencing factor nodes are retrieved from the causal graph. An intervention operation is performed on each influencing factor node, fixing its value, and the changes in the first and second offset judgment indices in the causal graph after the intervention are calculated. This process employs counterfactual reasoning, simulating the intervention to assess the causal effect of each influencing factor on the outcome and identify the most critical root cause.

[0192] Counterfactual reasoning is one of the core methods of causal inference. Its basic idea is to answer the question, "What if the situation were different?" In root cause analysis, counterfactual reasoning answers the question, "How would the deviation indicator change if a certain influencing factor remained at its normal value instead of its current outlier value?" By comparing the actual facts (abnormal influencing factor, indicator deviation) with the counterfactual assumptions (normal influencing factor, indicator change), the causal contribution of that influencing factor to the deviation can be quantified.

[0193] Intervention operations are a concrete implementation of counterfactual reasoning. In a causal graph, intervention operations are represented by the do operator. Performing an intervention on a node means forcibly setting the value of that node to a specific value and cutting off all causal edges pointing to that node, so that the value of that node is no longer affected by its cause, but is completely determined by external intervention. This simulates the situation of artificially controlling the variable in an experimental environment.

[0194] The system acquires the actual values ​​of all influencing factor nodes when the offset alarm occurs. These values ​​record the status of business variables at the time of the offset, where some variables may be in outlier or normal values. The system also needs to determine the normal or target value for each influencing factor node. The normal value can be calculated from historical normal data, such as using the historical mean, median, or mode, or it can be set according to business rules, such as setting the normal value of equipment temperature to the center value of the standard operating temperature range.

[0195] The system iterates through all influencing factor nodes and performs individual intervention analysis on each node. For the currently iterated influencing factor node, the system performs the following operations: First, it locates the node in the causal graph, records its current value as an outlier, and records its normal value as an intervention value. Then, the system performs the intervention operation, forcibly setting the value of the node to the intervention value and cutting off all causal edges pointing to the node, indicating that the value of the node is no longer affected by upstream causes but has been artificially fixed.

[0196] Forward reasoning is performed using a causal graph to calculate the value of the resulting node after intervention. Forward reasoning propagates along the causal edges from influencing factor nodes to the resulting node. For nodes that have been intervened in, their value is fixed as the intervention value; for nodes that have not been intervened in, their value is calculated from upstream nodes based on the causal relationship. The specific calculation method depends on the functional form of the causal edges. If the causal graph records the causal function of the edges, such as a linear relationship, exponential relationship, or other functional relationship, the system directly uses the function for calculation. For example, if sales equal base sales minus price multiplied by a price sensitivity coefficient, when the price is intervened in, sales are recalculated according to this function.

[0197] If the causal graph lacks an explicit functional form, the system can estimate the causal function from historical data using statistical regression methods. For each causal edge, the system extracts observed values ​​of the causal and outcome variables from historical data, fits a regression model (such as linear regression, multinomial regression, or neural network regression), and learns how the causes influence the outcomes. During intervention analysis, the fitted regression model is used for prediction, with the post-intervention values ​​of the causal variables as input and the predicted values ​​of the outcome variables as output.

[0198] The calculation is performed layer by layer, starting from the node of the influencing factor that has been intervened upon. The new values ​​of the downstream nodes directly affected by this factor are calculated, and then the new values ​​of the even downstream nodes affected by these downstream nodes are calculated. This process continues until the result node is reached, which represents the first and second offset determination indices. The resulting values ​​of the first and second offset determination indices after intervention represent the expected levels of the system's offset determination indices if the influencing factor remained normal.

[0199] Calculate the magnitude of change in the result nodes before and after the intervention. For the first offset indicator, the magnitude of change equals the value after the intervention minus the actual value before the intervention. If the magnitude of change is positive and large, it indicates that restoring the influencing factor to normal can significantly improve the first offset indicator, making it closer to the normal level, indicating that the influencing factor is an important reason for the decline in the first offset indicator. Similarly, calculate the magnitude of change in the second offset indicator to assess the impact of this influencing factor on population connectivity.

[0200] By combining the two magnitudes of change, a weighted sum or the maximum value can be used to obtain the combined magnitude of change. The combined magnitude of change reflects the strength of the causal effect of the influencing factor on the overall distribution shift. Record the combined magnitude of change corresponding to each influencing factor node to form a mapping table of influencing factors and magnitudes of change.

[0201] Repeat the above intervention analysis process for all influencing factor nodes to obtain complete causal effect assessment results. It is important to note that each intervention analysis is independent; that is, when intervening at node A, other nodes retain their actual observed values, and when intervening at node B, node A reverts to its actual observed values. This is a univariate intervention analysis, assessing the independent causal effect of each factor. Multivariate joint intervention analysis can also be performed, simultaneously intervening in multiple influencing factors to assess their synergistic effects, but the computational complexity increases exponentially. In practical applications, univariate analysis is usually performed first to identify key factors, followed by joint analysis of these key factors.

[0202] Next, all influencing factor nodes are sorted in descending order based on the magnitude of change. The top three influencing factor nodes with the largest magnitude of change are selected as the key root causes. The descending order places the factors with the strongest causal effect at the top. These factors contribute the most to the distribution shift and are the main causes of the problem, so they should be addressed first.

[0203] Obtain the comprehensive change magnitude of all influencing factor nodes calculated in step S15.3, and sort the influencing factor nodes using a sorting algorithm. The sorting key is the comprehensive change magnitude, and the sorting direction is descending, meaning the nodes with larger change magnitudes are listed first. The sorted list clearly shows the importance ranking of each influencing factor; the higher the ranking, the greater the impact of that factor on the offset, and the more significant the effect of fixing that factor on restoring normalcy.

[0204] Select the top 3 influencing factors from the ranking list and identify them as critical root causes. Choosing the top 3 is based on practical experience; in most scenarios, the core reasons for distribution shifts are usually no more than 3 to 5. Identifying these key causes and addressing them specifically will yield the greatest improvement. Choosing the top 3, rather than more, also aims to focus attention and resources, avoiding information overload and scattered efforts. This follows the Pareto principle (80 / 20 rule): concentrating on solving the most important 20% of problems can eliminate 80% of the impact.

[0205] In some cases, a dynamic thresholding method can be used instead of a fixed selection of the top three. For example, all influencing factors whose changes exceed a certain threshold can be selected as key root causes, or a set of influencing factors whose changes account for a certain proportion (such as 50% or 80%) of the total changes can be selected. This dynamic method can adapt to scenarios of varying complexity. In simple scenarios with a single root cause, only one or two key root causes may be selected, while in scenarios with complex root causes, four or five key root causes may be selected, thus more flexibly reflecting the actual situation.

[0206] The selected key root causes are validated and interpreted, checking whether they are consistent with the root causes of historical offset events. If the current offset event is similar to a historical event, and the identified key root causes are also consistent with historical root causes, this enhances the credibility of the analysis results. It is also checked whether the values ​​of the key root causes are indeed in an abnormal state. If a factor identified as a key root cause is actually within the normal range, this may indicate an error in the causal analysis, requiring manual review or adjustment of the causal graph structure.

[0207] Detailed causal explanations are generated for each key root cause, illustrating how it affects the offset assessment metrics. These explanations include visualizations of the causal path—the directed path from the root cause node to the outcome node—marking intermediate nodes and causal strength to help users understand the impact mechanism. The explanations also include quantitative analysis, showing how much the first offset assessment metric is expected to improve, the second offset assessment metric is expected to improve, and the overall offset status is expected to improve if the root cause is restored from its current outlier to a normal value. This provides a quantitative basis for developing corrective measures.

[0208] Finally, the key root causes are output to the visualization interface, yielding the root cause analysis results. The visualization interface serves as the window for user interaction with the system; through intuitive graphical displays, users can quickly understand the conclusions of the root cause analysis and make appropriate decisions.

[0209] The visualization interface design comprises multiple components, with the core component being the cause-and-effect graph visualization. The system displays the complete cause-and-effect graph structure in the form of a network diagram. Nodes are represented by different colors or shapes to indicate different types. Result nodes are represented by red or squares, key root cause nodes are highlighted in orange or stars, and other influencing factor nodes are represented by gray or circles. Edges use arrows to indicate the causal direction, and the thickness or color depth of the edges indicates the causal strength. The causal path from key root cause to result node is highlighted with a color for easy user identification.

[0210] The interface provides interactive features. Users can click on nodes to view detailed information, such as the name of the business variable the node represents, its current value, normal range, degree of anomaly, and historical trends. Users can click on edges to view detailed descriptions of causal relationships, such as the form of the causal function, causal strength, confidence level, and source (rule or data learning). Users can drag and drop nodes to adjust the graph layout, scale the graph, and filter the display of specific types of nodes or edges, allowing for flexible exploration of the causal graph.

[0211] The interface includes a critical root cause list component, displaying the top three critical root causes in a table format. Table columns include root cause name, current value, normal value, degree of deviation, causal effect strength (i.e., overall magnitude of change), and a summary of the causal path. The table supports clickable sorting, allowing users to sort by different columns and analyze root causes from different perspectives. The table also provides operation buttons, such as the "View Details" button to jump to the detailed root cause analysis page, the "Generate Report" button to export the root cause analysis report, and the "Develop Actions" button to initiate the corrective action recommendation process.

[0212] The interface includes a trend comparison chart component, displaying the time-series trends of key root cause variables. It compares the numerical changes during the offset period and the normal period, marking anomalous change points to help users understand when the root cause became abnormal and how this abnormality evolved. The trend chart can also be overlaid with intervention simulation curves, showing the expected recovery trajectories of the first and second offset criteria if intervention is implemented, providing predictive support for users to assess the effectiveness of interventions.

[0213] The interface includes a suggested action component. Based on the identified key root causes, it retrieves matching corrective actions from the action knowledge base and displays them to the user. The action knowledge base is a pre-built root cause-action mapping library that records standard handling procedures and best practices for various root causes. For example, actions for "equipment overheating" include "checking the cooling system," "reducing the load," and "cleaning the radiator." The actions are personalized based on the specific circumstances of the current root cause, such as recommending different priority combinations of actions based on the degree of temperature exceedance.

[0214] The interface also provides the ability to generate and export root cause analysis reports. The reports present all analysis results in a structured document format, including an overview of the off-events, a cause-effect diagram structure, a list of key root causes, quantitative analysis of causal effects, trend comparison charts, a list of recommended measures, and references to historical similar events. The reports can be exported as PDF, Word, or HTML formats, facilitating sharing with team members, management, or external auditors, as well as archiving and knowledge accumulation.

[0215] By enhancing causal root cause analysis, a leap from data-driven to knowledge-driven approaches has been achieved. By fully utilizing business logic rules and historical experience, combined with rigorous counterfactual reasoning methods, the key root causes of distribution deviations are accurately identified, providing scientific decision support for rapid problem solving and continuous improvement, and significantly improving operational efficiency and system reliability.

[0216] Example 12: In this embodiment, based on the business logic rules, a set of influencing factor variables and a set of result variables are defined to obtain an initial node set; based on the historical offset event data, a causal discovery algorithm is used to calculate the conditional independence between the influencing factor variables in the set of influencing factor variables and the result variables in the set of result variables, and the direction of the causal edges is determined to obtain a set of causal edges; the initial node set is used as nodes and the set of causal edges is used as directed edges to construct a directed acyclic graph to obtain the causal graph.

[0217] Specifically, firstly, based on business logic rules, a set of influencing factor variables and a set of outcome variables are defined to obtain an initial node set. This initial node set is the fundamental building block of the causal graph, clearly defining the scope of all variables involved in the causal analysis and providing a pool of candidate variables for subsequent causal edge identification.

[0218] The definition of the outcome variable set is relatively clear. According to the technical solution of this invention, the outcome variables are defined by two core metrics: a first offset criterion and a second offset criterion. The first offset criterion reflects the degree of distribution offset at the individual level. It is obtained by calculating the total probability value of the real-time data feature vector belonging to the normal distribution model. The lower the criterion, the more the individual data distribution deviates from the normal state. The second offset criterion reflects the distribution consistency at the group level. It is obtained by calculating the connectivity of the similarity network. The lower the criterion, the worse the consistency within the group, and the more likely there is a group offset. These two outcome variables are the targets of causal analysis, and it is necessary to identify which influencing factors cause their abnormal changes.

[0219] The definition of the influencing factor variable set is more complex, requiring the extraction of all business variables that may affect the outcome variable from the business logic rules. First, the business logic rules are parsed to identify all business variable names mentioned within them. Business logic rules are typically described in the form of condition-result or cause-effect, such as "When product inventory is below safety stock, the stockout rate increases," "When equipment load exceeds rated power, equipment temperature rises, increasing the risk of failure," and "When user traffic surges, server response time increases, degrading user experience." Natural language processing technology is then used to identify entities and relationships in the rule text, extracting business variables such as product inventory, stockout rate, equipment load, equipment temperature, user traffic, and server response time.

[0220] All extracted business variables are added to the influencing factor variable set, along with some general influencing factor variables. These variables, while not explicitly mentioned in the rules, typically impact the system's state based on domain knowledge and experience. For example, in e-commerce scenarios, variables such as price, sales volume, ratings, promotional indicators, holiday indicators, and competitor activities are added; in industrial scenarios, variables such as ambient temperature, humidity, raw material batches, and operator proficiency are added; and in financial scenarios, variables such as market indices, interest rates, exchange rates, and policy changes are added. These supplementary variables expand the coverage of causal analysis, avoiding the omission of important influencing factors.

[0221] The system deduplicates and normalizes the set of influencing factor variables to ensure that only one variable represents the same business concept. Variables from different sources using different names to represent the same concept are merged into a unified name. The system also categorizes and labels variables, marking their type (e.g., numeric, categorical, Boolean) and their business meaning (e.g., performance indicators, resource indicators, quality indicators, external factors). This metadata is helpful for subsequent causal analysis and result interpretation.

[0222] The set of outcome variables and the set of influencing factor variables are merged to obtain the initial node set. The initial node set contains all possible nodes in the causal graph, typically including dozens to hundreds of variables, the exact number depending on the business complexity and the comprehensiveness of rule coverage. The completeness of the node set directly affects the accuracy of causal analysis; if the true root cause variable is missed, subsequent analysis will be unable to identify the root cause; if too many irrelevant variables are included, it will increase computational complexity and noise interference. A balance must be struck between completeness and conciseness.

[0223] Secondly, based on historical offset event data, a causal discovery algorithm is used to calculate the conditional independence between the influencing factor variables in the influencing factor variable set and the outcome variables in the outcome variable set, determining the direction of the causal edges and obtaining the causal edge set. The causal edge set defines the causal relationships between variables, is the core structure of the causal graph, and determines the logical path of causal reasoning.

[0224] Conditional independence is the theoretical foundation for causal discovery. Its core idea is to infer causal structures by examining the independence relationships between variables given other variables. If variables A and B are statistically independent under all conditions, it indicates that there is no causal relationship between A and B, nor is there a common cause. If A and B are correlated unconditionally but conditionally independent given variable C, it indicates that C is a common cause of A and B, and there is no direct causal relationship between A and B. If A and B are correlated under all conditions, it indicates that there is a direct causal relationship between A and B, or that a causal path from A to B exists.

[0225] The PC algorithm is used as the causal discovery algorithm. PC is a classic example of constraint-based causal discovery methods, named after its inventors Peter and Clark. The PC algorithm consists of two main stages: a skeleton learning stage and an edge orientation stage. In the skeleton learning stage, it identifies whether edges exist between variables, regardless of edge direction, thus obtaining the skeleton of the undirected graph. In the edge orientation stage, it determines the causal direction of the edges, transforming the undirected edges into directed edges, resulting in a directed acyclic graph.

[0226] During the skeleton learning phase, a completely undirected graph is first constructed, meaning undirected edges are established between all pairs of nodes in the initial node set, assuming that a relationship may exist between any two variables. Then, the system progressively tests and removes edges that do not have a relationship, using conditional independence testing. For each pair of adjacent nodes, the system checks whether they are independent given different subsets of other nodes. If a set of conditions is found that makes the two nodes conditionally independent, it indicates that there is no direct causal relationship between them, and the undirected edge connecting them is removed.

[0227] The specific implementation of conditional independence tests depends on the variable type and data characteristics. For continuous numerical variables, the system uses partial correlation tests to calculate the correlation coefficient between the two variables after controlling for the influence of other variables. If the partial correlation coefficient is close to zero and statistically insignificant, the variables are considered conditionally independent. For discrete categorical variables, the system uses chi-square independence tests or G-tests to construct conditional contingency tables and test whether the frequency distributions of the two variables are independent under given conditions. For mixed-type variables, the system uses non-parametric methods such as kernel independence tests. These methods do not rely on distributional assumptions about the variables and have wider applicability.

[0228] Starting with an empty condition set, the system gradually increases the size of the condition set, performing conditional independence tests on each pair of nodes. When the condition set size is zero, it tests for unconditional independence, i.e., whether the two variables are inherently correlated. When the condition set size is one, it tests for conditional independence after controlling for one variable; the system iterates through all possible univariate condition sets. When the condition set size is two, it tests for conditional independence after controlling for two variables, and so on. The system sets an upper limit on the condition set size, typically between 3 and 5, to avoid computational explosion and statistical unreliability problems caused by excessively large condition sets.

[0229] After the skeleton learning phase, an undirected graph skeleton is obtained, in which edges connect all pairs of variables with some kind of dependency, but the causal direction has not yet been determined. The separation set of each pair of nodes is also recorded, which is the smallest set of variables that are conditionally independent; this separation set information will be used in the edge orientation phase.

[0230] In the edge orientation phase, a series of orientation rules are used to transform undirected edges into directed edges, determining the causal direction. The first rule is the v-structure recognition rule. A v-structure refers to a structure formed by three variables A, B, and C, where A and C are both adjacent to B, but A and C are not adjacent, and during skeleton learning, it was found that the separating set of A and C does not contain B. This structure can only be explained as a reason why A and C are both B, i.e., there are two causal edges from A to B and from C to B, forming the v-structure A→B←C. The system traverses all triples that satisfy the v-structure condition, orienting the corresponding undirected edges as directed edges pointing to intermediate nodes.

[0231] The second rule is the transitive orientation rule. If A→B is known, and there is an undirected edge BC between B and C, and A and C are not adjacent, then B→C can be deduced. This is because if it were C→B, it would form a v-structure of A→B←C, but A and C are not adjacent and do not satisfy the separation condition of the v-structure, creating a contradiction. Therefore, it can only be B→C. The system applies the transitive rule to derive the new edge direction using the already oriented edges.

[0232] The third rule is the acyclic orientation rule. If orienting an undirected edge in a certain direction would create a cycle in the graph, then that direction is not feasible and the edge must be oriented in the opposite direction. The system uses topological sorting or depth-first search to detect cycles, ensuring that the oriented graph retains its directed acyclic graph property. The system repeatedly applies these orientation rules until no more edges can be oriented.

[0233] After the edge orientation phase, some undirected edges may still exist in the graph that cannot be oriented. These edges are called uncertain edges, indicating that the causal direction cannot be uniquely determined from the data alone, and additional information such as time sequence or domain knowledge may be needed to resolve them. Uncertain edges can be marked as bidirectional edges or left undirected, and both possible directions of the uncertain edges can be considered in subsequent analysis, or the user can be prompted to make a manual judgment.

[0234] The system modifies and supplements causal edges based on business logic rules. If a causal edge discovered by the PC algorithm is consistent with a business rule, the confidence level of that edge is increased. If a causal edge discovered by the PC algorithm conflicts with a business rule, the system marks the conflict and prompts the user to review it, prioritizing the direction of the business rule because rules represent deterministic domain knowledge, while data learning may be affected by sample bias or noise. If a business rule describes a causal relationship but the PC algorithm fails to discover it, the system adds that edge to the causal edge set and marks its source as a rule, ensuring that important prior knowledge is not overlooked.

[0235] All identified directed edges are aggregated to form a causal edge set. This set is a list of edges, each recording attributes such as the starting node, ending node, causal strength, confidence level, and source. Causal strength can be estimated from the data, such as using the absolute value of conditional correlation coefficients, mutual information, or regression coefficients as a measure of strength; a higher causal strength indicates a more significant impact of the cause on the result. Confidence level reflects the reliability of the causal relationship. Edges representing the source of a rule have the highest confidence level, such as 1.0. Edges representing data learning have confidence levels set based on the p-value or confidence interval of a statistical test; a smaller p-value indicates a higher confidence level.

[0236] Finally, a directed acyclic graph (DAG) is constructed using the initial set of nodes as nodes and the set of causal edges as directed edges, resulting in a causal graph. A DAG is a graphical representation of causal relationships, clearly demonstrating the causal dependency network between variables through its graph structure, providing a formal foundation for subsequent causal inference and root cause analysis.

[0237] Create a graph data structure and store graph information using an adjacency list or adjacency matrix. For each node in the initial node set, the system creates a corresponding node object in the graph. The node object contains attributes such as node identifier, variable name, variable type, current value, and historical statistics. The system sets special markers for result variable nodes to distinguish them from influencing factor nodes, facilitating quick location of the analysis target in the graph.

[0238] For each edge in the causal edge set, the system creates a corresponding directed edge object in the graph. The directed edge connects the start node and the end node, with the direction pointing from cause to effect. The edge object contains attributes such as causal strength, confidence level, causal function, and source. If the causal function is known or can be fitted from the data, the system stores the function expression or model in the edge object for quantitative calculation during causal inference. If the causal function is unknown, the system marks it as undefined and uses statistical estimation methods during inference.

[0239] The system verifies whether the constructed graph satisfies the requirement of a directed acyclic graph (DAG), meaning there is no path from any node back to that node along a directed edge. The system uses a topological sorting algorithm for verification. Topological sorting attempts to arrange the nodes of the graph into a linear sequence such that for any directed edge from node A to node B, A always precedes B in the sequence. If the topological sort is successful, the graph is a DAG; if the topological sort fails, a cycle exists in the graph.

[0240] When a cycle is detected, it needs to be eliminated to meet the requirements of a directed acyclic graph (DAG). The system first identifies all edges in the cycle, analyzes the confidence level and source of each edge, and selects the edge with the lowest confidence level or the least reliable source for deletion or reversal to break the cycle. The system can prompt users to participate in the decision-making process, displaying the variables and causal relationships involved in the cycle, and asking users to use their business knowledge to determine which causal relationships might be erroneous or unimportant. Manual intervention ensures the rationality of the cycle elimination operation.

[0241] The constructed causal graph is optimized and simplified by removing isolated nodes, i.e., nodes without any causal edges connected to them, as these nodes do not contribute to causal analysis. Redundant edges are also removed. If there exists both a direct edge A→C and an indirect path A→B→C from node A to node C, and the causal effect of the direct edge can be fully explained by the indirect path, the system can remove the direct edge to simplify the graph structure, following the principle of simplicity, making the causal graph clearer and easier to understand.

[0242] A hierarchical structure is assigned to the causal graph, categorizing nodes according to the length of the causal chain. Nodes furthest from the outcome node are located at the bottom layer, called the root cause layer; these nodes are typically external inputs or original driving factors. Nodes closer to the outcome node are located in the middle layers, called the intermediary layers; these nodes are influenced by the root causes and further affect the outcome. The outcome node is located at the top layer, called the target layer. This hierarchical structure facilitates visualization and causal reasoning, with causal effects propagating from the bottom to the top during reasoning.

[0243] The completed causal graph is serialized and stored in JSON, XML, or graph database format, supporting persistence and cross-system sharing. As a core knowledge asset, the causal graph is continuously used during system operation. It is invoked for root cause analysis when distribution offset alarms occur, serves as a structural prior to improve prediction accuracy during trend prediction, and guides feature selection and model construction during parameter optimization, fully leveraging the value of causal knowledge.

[0244] Example 13: In this embodiment, hourly, daily, and weekly time windows are set, with the hourly window being 12 hours long, the daily window being 7 days long, and the weekly window being 4 weeks long, resulting in multi-scale time windows. For each multi-scale time window, the time series data and statistical feature change rate of the first and second offset judgment indicators within the window are extracted to obtain a time series feature set. Based on the historical time series feature set and the corresponding distribution offset labels, a long short-term memory network model is used to learn the mapping relationship between the time series feature set and the distribution offset labels, resulting in a trained trend prediction model. The current time series feature set is input into the trained trend prediction model, and the predicted probability value of the distribution offset occurring in the future time period is output. Based on the predicted probability value, a high-risk warning is issued when the predicted probability value is greater than 80%, a medium-risk warning is issued when the predicted probability value is between 50% and 80%, and a low-risk warning is issued when the predicted probability value is less than 50%, resulting in a graded warning result.

[0245] Specifically, hourly, daily, and weekly time windows are first set up, with the hourly window being 12 hours long, the daily window 7 days long, and the weekly window 4 weeks long, resulting in multi-scale time windows. This multi-scale time window design aims to capture data change patterns at different time granularities. Short-term windows capture instantaneous fluctuations and rapid changes, while long-term windows capture trend evolution and cyclical patterns. The comprehensive utilization of multi-scale information can improve the comprehensiveness and accuracy of trend prediction.

[0246] The three time windows cover multiple time scales from hours to months, forming a hierarchical time perspective. The short-term window focuses on immediate reactions, the medium-term window focuses on trend development, and the long-term window focuses on strategic situations. These three complement each other and together constitute a comprehensive time-dimensional analysis framework. In practical applications, the system maintains a sliding window for each time scale. The window is continuously updated over time, always containing the most recent data for the corresponding duration, ensuring that the analysis is always based on the latest situation.

[0247] Secondly, for each multi-scale time window, the time series data of the first and second offset criteria, as well as the rate of change of statistical features, are extracted to obtain the time series feature set. The time series feature set serves as the input to the trend prediction model, containing various statistical indicators and change characteristics describing historical evolution patterns, providing a rich source of information for model learning.

[0248] First, time series data is extracted. For each time window, the complete time series of the first and second offset judgment indicators within the window are obtained. The time series is a numerical sequence arranged in chronological order, recording the value of the indicator at each moment within the window. For hourly windows, if the data sampling frequency is once per minute, a 12-hour window contains 720 data points; if the sampling frequency is once per hour, it contains 12 data points. For daily and weekly windows, the original high-frequency data is usually downsampled, such as by hourly or daily aggregation, to reduce the amount of data while retaining the main trend information.

[0249] Extracting statistical features from time series data involves identifying numerical summaries of the distribution characteristics of the time series. Commonly used statistical features include the mean, standard deviation, maximum, minimum, median, and quartiles. The mean reflects the average level of the indicator within a window, the standard deviation reflects the degree of volatility, the maximum and minimum values ​​reflect extreme cases, and the median and quartiles reflect the quantile distribution. Calculating these statistical features forms a statistical feature vector, and the statistical features of different windows reflect the distribution characteristics at different time scales.

[0250] Extracting the rate of change feature of the time series, the rate of change describes the speed and direction of the indicator's change over time, serving as an important signal for trend prediction. The absolute rate of change and relative rate of change are calculated. The absolute rate of change equals the current value minus the initial value, reflecting the absolute amount of change within the window; the relative rate of change equals the absolute rate of change divided by the initial value, reflecting the relative magnitude of the change. The rate of change of the rate of change itself, i.e., the second derivative, is also calculated, describing the acceleration or deceleration of the trend. A positive second derivative indicates an accelerating downward trend and increasing risk; a negative second derivative indicates a decelerating downward trend and a potential bottoming out and rebound.

[0251] By extracting trend features from time series data and fitting linear or multinomial regression models, the slope and curvature parameters of the trend line are obtained. The slope reflects the overall direction and rate of change of the indicator; a positive slope indicates an upward trend, and a negative slope indicates a downward trend. The larger the absolute value of the slope, the faster the change. Curvature reflects the degree of bending of the trend; high curvature indicates a rapid trend reversal, while low curvature indicates a relatively stable trend. Trend features have direct predictive value for judging future trends.

[0252] The system extracts the volatility characteristics of time series data and describes the time-varying nature of the volatility by calculating the variance or standard deviation of the rolling window. It also calculates the number of peaks and troughs to describe the oscillation frequency of the series; frequent alternation of peaks and troughs indicates instability, while a smooth curve indicates stability. The system calculates the autocorrelation coefficient to describe the correlation of the series at different time lags; strong autocorrelation indicates that the series has memory, with past values ​​having a significant impact on the future, while weak autocorrelation indicates that the series is more random, and past values ​​have limited predictive value for the future.

[0253] All features extracted from the three time windows are aggregated to form a comprehensive temporal feature set. This temporal feature set is a high-dimensional feature vector containing dozens to hundreds of feature dimensions, comprehensively depicting the historical evolution pattern of the offset determination index across multiple time scales. Features are then normalized, scaling features with different dimensions and value ranges to a uniform scale, such as between 0 and 1, or standardizing them to a mean of 0 and a standard deviation of 1. Normalization eliminates the impact of dimensional differences on model training, accelerates model convergence, and improves prediction performance.

[0254] Next, based on the historical time-series feature set and the corresponding distribution offset labels, a Long Short-Term Memory (LSTM) network model is used to learn the mapping relationship between the time-series feature set and the distribution offset labels, resulting in a trained trend prediction model. LTM is a special type of recurrent neural network that effectively handles long-term dependency problems through gating mechanisms. It performs excellently in time series prediction tasks, capable of learning complex nonlinear mapping relationships from historical patterns to accurately predict future distribution offset risks.

[0255] First, the training dataset is prepared by extracting all labeled event records from the historical offset event database. For each event, the system obtains the temporal feature set of a window preceding the event as input samples, and whether the event ultimately experienced a distribution shift as the output label. The distribution shift label is a binary classification label: if a distribution shift occurred within a specific future time period, the label is 1 indicating a positive class; if no shift occurred, the label is 0 indicating a negative class. The temporal feature set of normal operation periods is also extracted as negative samples to ensure a balance between positive and negative samples in the training data, preventing the model from being biased towards the majority class.

[0256] The training dataset is divided into training, validation, and test sets, typically in a 70%, 15%, and 15% ratio. The training set is used for model parameter learning, the validation set for hyperparameter tuning and model selection, and the test set for final performance evaluation. It's crucial to pay attention to the chronological order of this division, ensuring that the validation and test sets are created later than the training set. This avoids using future data to predict past data, preventing data leaks and ensuring the accuracy of the evaluation.

[0257] A Long Short-Term Memory (LSTM) network model architecture is constructed, consisting of an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer receives a temporal feature set, with the feature dimension equal to the length of the temporal feature set. The LSTM layer is the core of the network, containing one or more LSTM unit layers. Each LSTM unit includes an input gate, a forget gate, an output gate, and a cell state. Through a gating mechanism, information is selectively retained and forgotten, enabling the modeling of long-term dependencies. The number of LSTM layers is typically 1 to 3, with each layer typically containing 50 to 200 units, depending on the task complexity and data size.

[0258] Fully connected layers perform non-linear transformations on the output of LSTM layers to extract high-level semantic features. They typically contain one or two fully connected layers, with the number of neurons in each layer gradually decreasing, resulting in feature compression and abstraction. The output layer is a single-neuron layer that uses the sigmoid activation function, outputting a probability value between 0 and 1, representing the predicted probability of a distribution shift. Dropout regularization is added between layers, randomly discarding some neuron connections to prevent overfitting and improve the model's generalization ability.

[0259] The loss function is defined as binary cross-entropy loss, which measures the difference between the model's predicted probability and the true label. The closer the predicted probability is to the true label, the smaller the loss. The system uses the Adam optimizer for parameter optimization. The Adam optimizer combines the advantages of the momentum method and the adaptive learning rate method, resulting in fast and stable convergence when training deep neural networks. The learning rate is typically set between 0.001 and 0.01. The learning rate determines the step size of parameter updates; too large a rate may lead to oscillations and non-convergence, while too small a rate may lead to slow convergence.

[0260] The model training process employs a mini-batch gradient descent method. Each time, a mini-batch of samples, typically containing 32 to 256 samples, is randomly selected from the training set and input into the model for forward propagation to calculate the predicted probabilities and loss. Then, backpropagation is used to calculate the gradient and update the model parameters. This process is repeated, traversing the entire training set; this is called one training epoch. The system typically trains for tens to hundreds of epochs until the model's performance on the validation set no longer improves.

[0261] After each training epoch, the model performance is evaluated on the validation set, and metrics such as validation loss, accuracy, precision, and recall are calculated to monitor the model's generalization ability. An early stopping strategy is adopted: if the validation loss does not decrease for several consecutive epochs, it indicates that the model is beginning to overfit, training is stopped, and the model parameters with the minimum validation loss are saved as the final model to avoid performance degradation caused by overtraining.

[0262] After training, the system evaluates the model's final performance on a test set. The test set data is completely unseen during training and can objectively reflect the model's predictive ability in real-world scenarios. The system calculates comprehensive metrics such as accuracy, precision, recall, F1 score, and area under the ROC curve on the test set. If the performance meets the preset targets, the model passes acceptance testing, is saved as a trained trend prediction model, and is deployed to the production environment.

[0263] Then, the current time-series feature set is input into the trained trend prediction model, which outputs the predicted probability value of distribution shifts occurring in future time periods. The predicted probability value is a quantitative assessment of future risks by the model, providing decision-makers with a clear risk metric and supporting risk management and resource allocation.

[0264] During real-time operation, the system continuously extracts the latest temporal feature set from the data stream using the same extraction method as during training, ensuring input format compatibility. The extracted temporal feature set is normalized using normalization parameters saved during training to ensure consistent feature scaling and avoid prediction errors caused by data distribution biases.

[0265] The trained trend prediction model is loaded. The model parameters have been learned and fixed during the training phase and will not be updated. The system inputs the normalized temporal feature set into the model. The feature vectors pass through the input layer to the LSTM layer. The LSTM layer performs temporal modeling based on the feature sequence and extracts time-dependent patterns. The output of the LSTM layer is passed to the fully connected layer for feature transformation. Finally, it passes through the sigmoid activation function of the output layer and outputs a value between 0 and 1, which is the predicted probability value.

[0266] The predicted probability value represents the likelihood of a distribution shift occurring within a specific future timeframe, given the historical evolution pattern described by the current time-series feature set. The length of the timeframe can be set according to business needs, such as the next 1 hour, 6 hours, 24 hours, or 7 days. Different timeframes correspond to different prediction models, and the system can train multiple models to predict risks for different timeframes, providing differentiated prediction information for different decision-making levels.

[0267] The system executes forecasts periodically, such as running the forecast process hourly, to update the forecast probability values ​​and reflect the latest risk situation. The system stores the forecast probability values ​​in a time-series database, recording information such as the forecast time, the target forecast period, and the forecast probability value, forming a historical forecast record for subsequent forecast performance evaluation and model optimization.

[0268] Finally, based on the predicted probability values, a high-risk warning is issued when the predicted probability value is greater than 80%, a medium-risk warning is issued when the predicted probability value is between 50% and 80%, and a low-risk warning is issued when the predicted probability value is less than 50%, resulting in a tiered warning system. This tiered warning system transforms continuous probability values ​​into discrete risk levels, making it easier for users to understand and make decisions. Different risk levels trigger different levels of response procedures.

[0269] A high-risk warning corresponds to a predicted probability value greater than 80%, meaning the model judges a very high likelihood of a distribution shift in the future, requiring immediate preventative measures. A high-risk warning triggers the highest-level response process, automatically notifying relevant personnel, including operations team leaders, business department heads, and technical experts, through channels such as phone calls, SMS messages, and emergency push notifications via instant messaging tools to ensure timely information delivery. Simultaneously, the system activates contingency plans, such as preparing backup resources, adjusting business strategies, increasing monitoring frequency, and suspending non-critical operations, to mitigate the impact of a potential shift.

[0270] A medium-risk warning corresponds to a predicted probability value between 50% and 80%, indicating a significant risk of deviation. Close monitoring and prepared contingency plans are necessary. A medium-risk warning triggers a medium-level response process, notifying operations and maintenance personnel and relevant technical staff via email and system messages. While immediate work interruption is not required, it should be prioritized in daily tasks. Operations and maintenance personnel are advised to conduct preventative checks, such as reviewing system logs, analyzing abnormal indicators, identifying potential problems, and preparing corrective action plans in advance to ensure a rapid response should the risk escalate.

[0271] A low-risk warning corresponds to a predicted probability value of less than 50%, meaning the likelihood of a short-term deviation is low and the system is in a relatively safe state, but routine monitoring is still necessary. Low-risk warnings typically do not trigger proactive notifications; the system records the risk information in the monitoring dashboard for periodic review by operations personnel. In a low-risk state, the system continues to execute routine monitoring and detection processes, tracking data changes and adjusting response strategies promptly should the risk level rise.

[0272] Visual indicators are set for each risk level, such as red for high risk, yellow for medium risk, and green for low risk. The current risk situation is clearly displayed on the monitoring interface using color coding. A risk trend graph is drawn, showing the curve of the predicted probability value changing over time, and the boundary lines of the risk levels are marked, so that users can intuitively see the direction of risk evolution and judge whether the risk is rising or falling, stable or fluctuating.

[0273] It supports customizable risk threshold configuration. Different business scenarios have different risk tolerance levels. Users can adjust the thresholds for high, medium, and low risks according to their actual needs. For example, they can adjust the high-risk threshold from 80% to 70% to improve the sensitivity of the warning, or adjust it to 90% to reduce the false alarm rate. After the threshold is adjusted, the system automatically re-evaluates historical prediction results and calculates the warning accuracy under the new threshold, helping users select the optimal threshold configuration.

[0274] The tiered early warning mechanism has achieved an upgrade from single alarms to risk quantification and tiered management, providing differentiated response strategies for different risk levels. It avoids resource waste caused by overreaction and prevents risk escalation caused by underreaction, truly realizing refined risk management and forward-looking prevention and control, and providing strong protection for business continuity and system stability.

[0275] Example 14: In this embodiment, a Long Short-Term Memory (LSTM) network model architecture is constructed, comprising an input layer, an LSM layer, a fully connected layer, and an output layer, to obtain an initial trend prediction model. The historical time-series feature set is used as input, and the distribution offset label is used as a supervision signal. The weight parameters of the initial trend prediction model are trained using a backpropagation algorithm. The cross-entropy loss between the predicted probability value output by the initial trend prediction model and the distribution offset label is minimized to obtain the training loss value. When the training loss value converges, the weight parameters of the initial trend prediction model are saved, resulting in the trained trend prediction model.

[0276] Specifically, firstly, a Long Short-Term Memory (LSTM) network model architecture is constructed. This architecture includes an input layer, an LTM layer, a fully connected layer, and an output layer, resulting in an initial trend prediction model. The model architecture defines the network structure and information flow pattern; it serves as the blueprint for the deep learning model and determines its expressive and learning capabilities.

[0277] The input layer is the first layer of the model, responsible for receiving the temporal feature set as external input. The dimension of the input layer is equal to the number of features in the temporal feature set; if the temporal feature set contains 100 features, the input layer will contain 100 input nodes. The input layer does not perform any computational transformations; it merely serves as the data entry point, passing feature values ​​to the next layer. A batch normalization layer can be added after the input layer to normalize each mini-batch of input, ensuring that each feature has a mean of 0 and a standard deviation of 1 within the batch. Batch normalization accelerates model training convergence and improves numerical stability.

[0278] The Long Short-Term Memory (LSTM) layer is the core component of the model, responsible for extracting time-dependent patterns and long-term memory information from temporal features. The LSTM layer consists of multiple LSTM units, each a complex computational unit containing five key components: cell state, hidden state, input gate, forget gate, and output gate. The cell state serves as the long-term memory carrier of the LSTM, persisting throughout the time series and selectively retaining important information while forgetting irrelevant information. The hidden state is the short-term output of the LSTM, updated at each time step and passed to the next layer or the next time step.

[0279] The input gate controls the degree to which new input information is added to the cell state. It calculates a threshold value between 0 and 1 based on the current input and the previous hidden state. A threshold value close to 1 indicates that new information is fully allowed to be added, while a value close to 0 indicates that new information is blocked. The forget gate controls the degree to which historical information is retained in the cell state. It also calculates a threshold value between 0 and 1, with a value close to 1 indicating that historical information is fully retained and a value close to 0 indicating that historical information is completely forgotten. The output gate controls the degree to which the cell state outputs information to the hidden state, adjusting the amount of information transmitted externally at the current time step.

[0280] Setting the number of Long Short-Term Memory (LSTM) layers and the number of units per layer is a crucial hyperparameter choice. The number of layers determines the model's depth; a single-layer LSTM is suitable for simple temporal patterns, while a multi-layer LSTM can learn more complex hierarchical features, but it also increases computational complexity and the risk of overfitting. Systems typically use 1 to 3 LSTM layers, adjusted based on task complexity and data volume. The number of units per layer determines the model's width; more units result in a larger model capacity, enabling the representation of more complex functions, but also requiring more training data and computational resources. Systems typically use 50 to 200 units per layer, striking a balance between expressive power and computational efficiency.

[0281] Fully connected layers further extract features and perform nonlinear transformations on the output of the LSTM layers, mapping the temporal features learned by the LSTM to the representation space required for the prediction task. Each neuron in a fully connected layer is fully connected to all neurons in the previous layer, achieving complex nonlinear mappings through weight matrices and activation functions. The system typically uses one or two fully connected layers. The first fully connected layer has a number of neurons comparable to or slightly fewer than the output dimension of the LSTM layer, used for feature integration. The second fully connected layer has a further reduced number of neurons, used for feature compression and abstraction, forming a pyramid-like structure.

[0282] The fully connected layers use the ReLU activation function. The ReLU function outputs the same value as the input when the input is greater than zero, and outputs zero when the input is less than or equal to zero. This non-linear activation introduces non-linear modeling capabilities while being computationally simple and efficient, mitigating the vanishing gradient problem. Dropout layers are added between the fully connected layers. Dropout randomly discards the activation values ​​of some neurons with a certain probability, forcing the model to be independent of specific neuron combinations, thus enhancing the model's robustness and generalization ability. The dropout probability is typically set to 0.3 to 0.5.

[0283] The output layer is the last layer of the model, responsible for outputting the final prediction result. For binary classification prediction tasks, the output layer contains one neuron, using the sigmoid activation function. The sigmoid function maps any real number to the range of 0 to 1, and the output value can be interpreted as the predicted probability, that is, the probability that the model believes the sample belongs to the positive class. The output layer receives the output of the fully connected layer, undergoes weight transformation and sigmoid activation, and produces the predicted probability value.

[0284] The system connects the layers in the order of input layer, long short-term memory layer, fully connected layer, and output layer to form a complete network architecture, creating the initial trend prediction model. All weight parameters of the initial model are randomly initialized, typically using Xavier or He initialization methods. These methods set appropriate initial weight distributions based on the input and output dimensions of the layers, avoiding training difficulties caused by initial weights being too large or too small. The initial model architecture is determined, but the parameters are not trained and it does not yet possess predictive ability; the parameters need to be learned through the training process.

[0285] Second, using historical time-series feature sets as input and distribution offset labels as supervisory signals, the backpropagation algorithm is employed to train the weight parameters of the initial trend prediction model. The cross-entropy loss between the predicted probability value output by the initial trend prediction model and the distribution offset labels is minimized to obtain the training loss value. Model training is the core process of deep learning. Through supervised learning with a large amount of labeled data, the model gradually adjusts its parameters, making the predicted output increasingly closer to the true labels, ultimately mastering the mapping pattern from input to output.

[0286] Backpropagation is a standard method for training neural networks. Its basic principle is to first perform forward propagation to calculate the model output and loss function value, then perform backpropagation to calculate the gradient of the loss function with respect to the parameters of each layer, and finally update the parameters based on the gradient. Through multiple iterations, the loss function is gradually reduced, and the model performance is gradually improved.

[0287] In the forward propagation phase, a mini-batch of samples is drawn from the training set, and the temporal feature set of these samples is input into the model. The data sequentially passes through the input layer, long short-term memory layer, fully connected layer, and output layer. Each layer performs calculations based on the current weight parameters, and the final output layer generates predicted probability values. The predicted probability values ​​are then compared with the true distribution offset labels of the corresponding samples, and the cross-entropy loss function value is calculated.

[0288] The cross-entropy loss function is the standard loss function for binary classification tasks, measuring the difference between the predicted probability distribution and the true label distribution. For a single sample, if the true label is 1 (positive class), the cross-entropy loss equals the negative logarithm of the predicted probability; the closer the predicted probability is to 1, the smaller the loss, and the further the predicted probability is from 1, the larger the loss. If the true label is 0 (negative class), the cross-entropy loss equals negative 1 minus the logarithm of the predicted probability; the closer the predicted probability is to 0, the smaller the loss, and the further the predicted probability is from 0, the larger the loss. For a mini-batch of samples, the total loss is the average of the losses of each sample, and the system obtains the training loss value for the current mini-batch.

[0289] During the backpropagation phase, the gradient of the loss function with respect to the parameters of each layer of the model is calculated. The gradient represents the degree of influence of a small change in the parameters on the loss function. The direction of the gradient is the direction in which the loss function increases the fastest, and the opposite direction is the direction in which the loss function decreases the fastest. Starting from the output layer, the gradient is calculated layer by layer using the chain rule. The gradient of the output layer is directly obtained from the derivative of the loss function with respect to the output, and then passed to the fully connected layer. The fully connected layer calculates the gradient based on its activation function and weights and continues to pass it forward. The gradient calculation of the LSTM layer is more complex, requiring consideration of time unfolding and gating mechanisms. The system uses the special gradient calculation formula of LSTM to finally obtain the gradients of all parameters.

[0290] During the parameter update phase, the optimizer updates the parameters based on the gradients, implementing specific parameter update rules. The Adam optimizer is a commonly used optimizer that combines momentum and adaptive learning rate techniques. The momentum method accumulates an exponentially weighted moving average of historical gradients, making the parameter update direction smoother and more stable, reducing oscillations, and accelerating convergence. The adaptive learning rate method adjusts the learning rate individually for each parameter, using a smaller learning rate for parameters with drastic gradient changes and a larger learning rate for parameters with gentle gradient changes, improving training efficiency and stability.

[0291] Setting the learning rate hyperparameter controls the step size of each parameter update. An excessively large learning rate may lead to training instability or even divergence, while a too small learning rate results in slow convergence. The system typically starts with 0.001 and adjusts the learning rate based on changes in loss during training. If the loss decreases slowly, the learning rate can be increased; if the loss fluctuates or increases, the learning rate needs to be decreased. The system can also employ a learning rate decay strategy, gradually reducing the learning rate as training progresses. A larger learning rate is used early on to quickly approach the optimal solution, while a smaller learning rate is used later for fine-tuning, improving the final performance.

[0292] Repeating the cycle of forward propagation, loss calculation, backpropagation, and parameter updates, traversing the entire training set, is called a training epoch. The system typically trains for tens to hundreds of epochs. After each epoch, the system evaluates the model's performance on the validation set, records the validation loss and accuracy, monitors the model's generalization ability, and prevents overfitting. During training, the loss value usually decreases as training progresses, decreasing rapidly initially, then gradually leveling off, and eventually stabilizing.

[0293] Third, when the loss value converges during the training process, save the weight parameters of the initial trend prediction model to obtain the completed trend prediction model. Loss value convergence means that the loss value no longer decreases significantly and reaches a relatively stable state, indicating that the model has fully learned the pattern of the training data, and the performance improvement brought by continuing training is limited. At this time, training should be stopped and the model saved.

[0294] The criteria for determining convergence are: after several consecutive training iterations (e.g., 5 to 10 iterations), the validation loss no longer decreases or decreases by a less than a preset threshold (e.g., 0.001), or the preset maximum number of training iterations is reached. An early stopping strategy is employed, continuously monitoring the validation loss during training. Once the validation loss begins to rise, indicating that the model is overfitting the training set and its generalization ability is declining, the system immediately stops training and reverts to the model state with the minimum validation loss, which is the moment when the model's generalization ability is strongest.

[0295] The model's weight parameters, including weight matrices, bias vectors, and normalization parameters for all layers, are saved. These parameters are serialized into file formats such as HDF5 or PyTorch's pt format and stored in a persistent storage system. The model's architecture configuration is also saved, recording structural information such as layer type, number, number of units, and activation functions, as well as metadata such as hyperparameters used during training, including learning rate, batch size, and training epochs, facilitating model reproduction and deployment.

[0296] The trained trend prediction model is a mature model that has been fully trained and validated. It has learned the mapping pattern from time-series feature sets to distribution offset probabilities and has the ability to predict future risks. Deploying the model to the production environment for real-time trend prediction and risk warning provides intelligent support for business decisions, realizing a complete closed loop of learning patterns from historical data and applying them to future predictions.

[0297] Example 10: like Figure 2 As shown, the present invention also provides an adaptive threshold detection system for multi-dimensional distribution offset, comprising: Feature extraction module 10 is used to acquire real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors; The normal distribution model training module 20 is used to train a Gaussian mixture model based on historical normal data using the expectation-maximization algorithm, determine the parameters of the Gaussian mixture model, and obtain the normal distribution model. The first offset determination module 30 is used to input the feature vector into the normal distribution model, calculate the probability value of the feature vector belonging to the normal distribution, and obtain the first offset determination index. The inter-object similarity calculation module 40 is used to calculate the distribution difference between any two detected objects based on the real-time data distribution of multiple detected objects within the same group, and obtain the inter-object similarity. The second offset determination module 50 is used to construct a similarity network based on the similarity between the objects, calculate the connectivity of the similarity network, and obtain a second offset determination index; wherein, the similarity network uses the detected object as a node and establishes edges between object pairs with a similarity greater than a preset similarity threshold; The adaptive threshold update module 60 is used to set a sliding window of fixed length and dynamically update the first offset judgment index and the second offset judgment index within the sliding window using an exponentially weighted moving average algorithm to obtain the first adaptive threshold and the second adaptive threshold. The distribution offset determination module 70 is used to determine that a distribution offset has occurred and output a distribution offset alarm when the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold. The model incremental update module 80 is used to update the parameters of the Gaussian mixture model in real time based on the normal data within the sliding window using the incremental expectation-maximization algorithm, so as to obtain the updated normal distribution model.

[0298] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An adaptive threshold detection method for multi-dimensional distribution offset, characterized in that, include: Acquire real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors; Based on historical normal data, the Gaussian mixture model is trained using the expectation-maximization algorithm, the parameters of the Gaussian mixture model are determined, and the normal distribution model is obtained. The feature vector is input into the normal distribution model, and the probability value of the feature vector belonging to the normal distribution is calculated to obtain the first offset judgment index. Based on the real-time data distribution of multiple detected objects within the same group, the Wasserstein distance is used to calculate the distribution difference between any two detected objects to obtain the similarity between objects. A similarity network is constructed based on the similarity between the objects, and the connectivity of the similarity network is calculated to obtain a second offset determination index; wherein, the similarity network uses the detected object as a node and establishes edges between object pairs with a similarity greater than a preset similarity threshold; A fixed-length sliding window is set, and the first offset judgment index and the second offset judgment index within the sliding window are dynamically updated using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold. When the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold, a distribution offset is determined to have occurred, and a distribution offset alarm is output. Based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm to obtain the updated normal distribution model.

2. The method according to claim 1, characterized in that, The step of extracting multi-dimensional statistical features from the real-time data to obtain a feature vector includes: Based on the real-time data, the arithmetic mean of the dataset is calculated to obtain the mean characteristic; Based on the real-time data and the mean feature, the degree of dispersion of the data deviating from the mean feature is calculated to obtain the standard deviation feature; Based on the real-time data and the mean characteristic, the degree of asymmetry in the data distribution is calculated to obtain the skewness characteristic; Based on the real-time data and the mean characteristic, the kurtosis of the data distribution is calculated to obtain the kurtosis characteristic; The real-time data is sorted, and the 25th percentile, 50th percentile, and 75th percentile are calculated to obtain the quartile features; The mean feature, standard deviation feature, skewness feature, kurtosis feature, and quartile feature are combined to obtain the feature vector.

3. The method according to claim 1, characterized in that, The process of training a Gaussian mixture model using the expectation-maximization algorithm based on historical normal data, determining the parameters of the Gaussian mixture model, and obtaining a normal distribution model includes: Based on the historical normal data, initialize the number of sub-distributions, the initial weights of each sub-distribution, the initial mean vector, and the initial covariance matrix of the Gaussian mixture model. Execute the expectation step to calculate the posterior probability of each data point in the historical normal data belonging to each sub-distribution, and obtain the membership matrix. Based on the membership matrix and the historical normal data, a maximization step is performed to update the weights, mean vectors, and covariance matrices of each sub-distribution, resulting in updated Gaussian mixture model parameters. Based on the updated Gaussian mixture model parameters, the log-likelihood function value of the historical normal data is calculated, and it is determined whether the log-likelihood function value has converged, thus obtaining the convergence determination result. When the convergence determination result is non-convergence, the expectation step and the maximization step are repeatedly executed based on the updated Gaussian mixture model parameters until the log-likelihood function value converges, and the final Gaussian mixture model parameters are obtained. The parameters of the final Gaussian mixture model are used as the model parameters of the normal distribution model to obtain the normal distribution model.

4. The method according to claim 1, characterized in that, The step of inputting the feature vector into the normal distribution model, calculating the probability value of the feature vector belonging to the normal distribution, and obtaining the first offset determination index includes: Obtain the weights, mean vectors, and covariance matrices of each sub-distribution from the normal distribution model, substitute the feature vectors into the probability density functions of each sub-distribution, and calculate the probability density values ​​of the feature vectors under each sub-distribution. The probability density values ​​under each sub-distribution are multiplied by the weights of the corresponding sub-distributions and then summed to obtain the total probability value of the feature vector belonging to the normal distribution model; The total probability value is used as the first offset determination index to obtain the first offset determination index.

5. The method according to claim 1, characterized in that, The method, based on the real-time data distribution of multiple detected objects within the same group, uses Wasserstein distance to calculate the distribution difference between any two detected objects to obtain the object similarity, including: Obtain the real-time data distribution of the first detection object and the real-time data distribution of the second detection object, represent the real-time data distribution of the first detection object as a first probability distribution, and represent the real-time data distribution of the second detection object as a second probability distribution to obtain the first probability distribution and the second probability distribution; Based on the first probability distribution and the second probability distribution, the minimum transmission cost to transform the first probability distribution into the second probability distribution is calculated, and the Wasserstein distance is obtained. Based on the Wasserstein distance, the object similarity between the first detected object and the second detected object is calculated using the formula S=1 / (1+W).

6. The method according to claim 1, characterized in that, The process of constructing a similarity network based on the similarity between the objects, calculating the connectivity of the similarity network, and obtaining a second offset determination index includes: Each detected object within the same group is set as a node in the similarity network to obtain a node set; Traverse any two nodes in the node set. When the similarity between the objects corresponding to the two nodes is greater than the preset similarity threshold, establish an edge between the two nodes to obtain the similarity network. Based on the similarity network, a graph theory algorithm is used to identify connected clusters in the similarity network, and the connectivity is obtained by calculating the proportion of the number of nodes in the largest connected cluster to the total number of nodes in the node set. The connectivity is used as the second offset determination index to obtain the second offset determination index.

7. The method according to claim 1, characterized in that, The method of dynamically updating the first offset determination index and the second offset determination index within the sliding window using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold includes: Obtain the first offset determination index at the current time and the first adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the first adaptive threshold at the current time. Obtain the second offset determination index at the current time and the second adaptive threshold at the previous time, set the smoothing coefficient α to 0.3, and calculate the second adaptive threshold at the current time.

8. The method according to claim 1, characterized in that, The duration of the sliding window is 24 hours.

9. The method according to claim 1, characterized in that, When the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold, a distribution offset is determined to have occurred, and a distribution offset alarm is output, including: Obtain the first offset determination index and the first adaptive threshold, determine whether the first offset determination index is lower than the first adaptive threshold, and obtain the first determination result; Obtain the second offset determination index and the second adaptive threshold, determine whether the second offset determination index is lower than the second adaptive threshold, and obtain the second determination result; When the first determination result is true or the second determination result is true, the current detection object is marked as having a distribution shift, and the distribution shift alarm is generated.

10. The method according to claim 1, characterized in that, Based on the normal data within the sliding window, the parameters of the Gaussian mixture model are updated in real time using the incremental expectation-maximization algorithm to obtain the updated normal distribution model, including: Data that are both higher than the first adaptive threshold and higher than the second adaptive threshold in the sliding window are selected to obtain incremental training data. Based on the incremental training data and the parameters of the current Gaussian mixture model, an incremental expectation step is performed to calculate the posterior probability of each data point in the incremental training data belonging to each sub-distribution, thereby obtaining the incremental membership matrix. Based on the incremental membership matrix and the incremental training data, the incremental maximization step is performed to update the weights, mean vectors and covariance matrices of each sub-distribution, and to obtain candidate update parameters. Based on the candidate update parameters, calculate the rate of change of the mean vector in the candidate update parameters relative to the mean vector in the parameters of the current Gaussian mixture model, determine whether the rate of change is less than or equal to 10%, and obtain the stability determination result. When the stability determination result is true, the candidate update parameters are used to replace the parameters of the current Gaussian mixture model to obtain the updated normal distribution model.

11. The method according to claim 1, characterized in that, Also includes: Obtain business logic rules and historical offset event data, and construct a directed acyclic graph based on the business logic rules and historical offset event data to obtain a causal graph; wherein, the result nodes of the causal graph are the first offset judgment index and the second offset judgment index, and the influencing factor nodes are business variables; When the distribution offset alarm is output, all influencing factor nodes are obtained from the causal graph, an intervention operation is performed on each influencing factor node, the value of the influencing factor node is fixed, and the change range of the first offset judgment index and the second offset judgment index in the causal graph after the intervention is calculated. Based on the magnitude of change, all the influencing factor nodes are sorted in descending order, and the top 3 influencing factor nodes with the largest magnitude of change are selected to obtain the key root causes; The key root causes are output to a visualization interface to obtain the root cause analysis results.

12. The method according to claim 11, characterized in that, The construction of a directed acyclic graph based on the business logic rules and the historical offset event data, resulting in a causal graph, includes: Based on the aforementioned business logic rules, define the set of influencing factor variables and the set of result variables to obtain the initial node set; Based on the historical offset event data, a causal discovery algorithm is used to calculate the conditional independence between the influencing factor variables in the influencing factor variable set and the outcome variables in the outcome variable set, determine the direction of the causal edge, and obtain the causal edge set; The initial set of nodes is used as nodes, and the set of causal edges is used as directed edges to construct a directed acyclic graph, thus obtaining the causal graph.

13. The method according to claim 1, characterized in that, Also includes: Set hourly, daily, and weekly time windows, wherein the length of the hourly time window is 12 hours, the length of the daily time window is 7 days, and the length of the weekly time window is 4 weeks, to obtain a multi-scale time window; For each of the multi-scale time windows, extract the time series data of the first offset determination index and the second offset determination index within the window, as well as the rate of change of statistical features, to obtain a time series feature set; Based on the historical time series feature set and the corresponding distribution offset label, a long short-term memory network model is used to learn the mapping relationship between the time series feature set and the distribution offset label, so as to obtain the trained trend prediction model. Input the current time series feature set into the trained trend prediction model, and output the predicted probability value of the distribution shift occurring in the future time period; Based on the predicted probability value, a high-risk warning is issued when the predicted probability value is greater than 80%, a medium-risk warning is issued when the predicted probability value is between 50% and 80%, and a low-risk warning is issued when the predicted probability value is less than 50%, thus obtaining a graded warning result.

14. The method according to claim 13, characterized in that, The process involves using a long short-term memory network model to learn the mapping relationship between the historical time-series feature set and the corresponding distribution offset labels, based on the historical time-series feature set and the corresponding distribution offset labels, to obtain a trained trend prediction model, including: A long short-term memory network model architecture is constructed, which includes an input layer, a long short-term memory layer, a fully connected layer, and an output layer, to obtain an initial trend prediction model; Using the historical time series feature set as input and the distribution offset label as a supervision signal, the weight parameters of the initial trend prediction model are trained using the backpropagation algorithm. The cross-entropy loss between the predicted probability value output by the initial trend prediction model and the distribution offset label is minimized to obtain the training process loss value. When the loss value of the training process converges, the weight parameters of the initial trend prediction model are saved to obtain the completed trend prediction model.

15. An adaptive threshold detection system for multi-dimensional distribution offset, characterized in that, include: The feature extraction module is used to acquire real-time data, extract multi-dimensional statistical features from the real-time data, and obtain feature vectors. The normal distribution model training module is used to train a Gaussian mixture model based on historical normal data using the expectation-maximization algorithm, determine the parameters of the Gaussian mixture model, and obtain the normal distribution model. The first offset determination module is used to input the feature vector into the normal distribution model, calculate the probability value of the feature vector belonging to the normal distribution, and obtain the first offset determination index. The inter-object similarity calculation module is used to calculate the distribution difference between any two detected objects based on the real-time data distribution of multiple detected objects within the same group, and obtain the inter-object similarity. The second offset determination module is used to construct a similarity network based on the similarity between the objects, calculate the connectivity of the similarity network, and obtain a second offset determination index; wherein, the similarity network uses the detected object as a node and establishes edges between object pairs with a similarity greater than a preset similarity threshold; An adaptive threshold update module is used to set a sliding window of fixed length and dynamically update the first offset judgment index and the second offset judgment index within the sliding window using an exponentially weighted moving average algorithm to obtain a first adaptive threshold and a second adaptive threshold. The distribution offset determination module is used to determine that a distribution offset has occurred and output a distribution offset alarm when the first offset determination index is lower than the first adaptive threshold or the second offset determination index is lower than the second adaptive threshold. The model incremental update module is used to update the parameters of the Gaussian mixture model in real time based on the normal data within the sliding window using the incremental expectation-maximization algorithm, so as to obtain the updated normal distribution model.

Images