A concept drift detection method and system based on adaptive data driving

Abstract

The present invention relates to a concept drift detection method and system based on adaptive data driving, which belongs to the field of machine learning technology. The method comprises the following steps: Step 1: pre-processing water purification plant data and setting prototype neural network parameters; Step 2: adjusting data distribution: extracting data distribution features from historical data and generating an initial data set in combination with resampling probability; Step 3: data variation, introducing three variation factors, generating a re-emergence data set; Step 4: for the re-emergence data set, obtaining meta-features; Step 5: dividing the re-emergence data set into a support set and a query set and inputting them into the prototype neural network for pre-training to obtain a trained prototype neural network; Step 6: inputting the pre-processed real-time water purification plant data into the trained prototype neural network for concept drift detection. Comprehensively considering periodic and non-periodic concept drift scenarios, the drift detection and response capabilities in complex environments are improved.

Description

Technical Field

[0001] The present invention relates to a concept drift detection method and system based on adaptive data driving, belonging to the technical field of machine learning. Background Art

[0002] Water quality testing at water treatment plants is a core component of ensuring drinking water safety. Through systematic, comprehensive monitoring of source water, treatment process water, and outlet water, water quality and process operation risks are effectively assessed. Testing indicators cover physical (such as turbidity and temperature), chemical (such as pH, residual chlorine, and total hardness), and nutrients (such as total nitrogen, total phosphorus, and ammonia nitrogen), and can be expanded to include organic pollutants, heavy metals, and microbial indicators, enabling comprehensive monitoring of water pollution loads. In recent years, the integration of the Internet of Things, big data, and artificial intelligence technologies has driven the development of online water quality monitoring systems. These systems utilize multi-source sensors for high-frequency, real-time data acquisition. Combined with intelligent analysis models, they can dynamically track water quality changes, provide early warning of pollution incidents, and assist in the intelligent adjustment of process parameters. Based on this, water treatment plants have gradually established an intelligent, closed-loop drinking water safety system, from source water protection to outlet water distribution, effectively ensuring the efficient, safe, and sustainable operation of urban water supply.

[0003] In machine learning operations (MLOps), concept drift has become a critical issue affecting model stability and robustness. Concept drift refers to the non-stationary evolution of the data generation mechanism over time, causing the relationship between features learned during training and target variables to shift in actual deployment, leading to degraded model performance or even failure. Traditional machine learning assumes that the distribution of training data and inference data is consistent, making it difficult to cope with data distribution changes caused by environmental perturbations, device aging, and changes in user behavior in complex application scenarios such as dynamic water quality monitoring. Drift can manifest as changes in prior probabilities, conditional distributions, or marginal distributions, impacting the model's discriminative capabilities to varying degrees. Therefore, building intelligent systems capable of continuous monitoring, automatic diagnosis, and dynamic adaptation is key to ensuring the long-term effectiveness and reliability of models. In-depth research on automated detection and adaptive mechanisms for concept drift has become an important direction for the full lifecycle management and optimization of machine learning systems, especially in highly dynamic data environments.

[0004] Existing research on the concept drift problem has typically adopted a two-stage "detection-adaptation" paradigm: first, statistical methods are used to identify changes in data distribution, followed by adaptation through strategies such as model retraining, integration of historical and current models, or incremental fine-tuning. This paradigm assumes that recent data better reflects the current state of concepts than historical data, but it suffers from two major limitations. First, detection relies heavily on pre-defined statistical tests and requires prior knowledge of data distribution, making it difficult to cope with the temporal dynamics and non-stationarity of data streams, resulting in unstable detection. Second, most methods can only determine whether drift has occurred in a binary manner, making it difficult to distinguish between sudden, gradual, periodic, and incremental drift types. This limits the refinement of adaptation strategies and reduces the efficiency and robustness of system response. Therefore, there is an urgent need to develop detection and adaptation mechanisms with stronger generalization capabilities and the ability to identify drift types to improve the long-term effectiveness and reliability of models in dynamic environments.

[0005] In recent years, prototype networks, a few-shot learning framework based on metric learning, have become a research hotspot for concept drift detection due to their ability to construct a highly interpretable feature space. This approach learns prototype vectors for each category and uses the distance between query samples and the prototypes for classification, enhancing its adaptability to new categories and few-shot classifications. However, in few-shot environments, data is limited and samples are highly correlated, and the supervisory information provided by query samples is scarce, limiting the model's generalization performance. Especially when the support set is noisy, category prototypes can easily deviate from the true distribution, affecting classification accuracy. In dynamic data streams, data distribution and concepts are constantly evolving, making static prototypes difficult to adapt to time-varying characteristics and resulting in insufficient adaptability. Improving the prototype network's ability to model dynamic changes in prototype positions and enhancing its sensitivity and adaptability to changes in data distribution and concept drift are key challenges that need to be addressed. This will effectively improve the model's robustness and detection accuracy in complex dynamic environments, and promote the development and application of concept drift detection technology. Summary of the Invention

[0006] In view of the shortcomings of existing technologies in concept drift detection in water quality data, the present invention proposes a concept drift detection pre-training method and system based on adaptive data-driven. In view of the continuity and regular evolution characteristics of periodic concept drift in the feature space, the present invention first models the historical observation data and generates a new type of data set by predicting the distribution of future data. The present invention defines it as an "initial classification data set". This data set aims to enhance the model's perception and adaptability to periodic concept drift. On this basis, a disturbance factor that simulates sudden changes is introduced to construct a "re-sudden data set" to simulate non-periodic drift situations, and it is used in the training process of the prototype network.

[0007] Through the pre-training mechanism based on meta-learning, the prototype network has the ability to adapt to unpredictable drift patterns such as suddenness and gradualness. In response to the problem that the class prototypes generated by traditional prototype networks under low-quality support sets are not representative enough, the present invention introduces a prototype interactive distillation mechanism to improve the robustness and discriminability of prototype expression, thereby reducing the impact of poor support set quality on model performance. In addition, the present invention further introduces a dynamic evolution factor based on data distribution in the prototype generation process to alleviate the problem of uneven data distribution in stream data processing and enhance the model's adaptability and generalization ability to dynamic changes in data distribution. This method comprehensively considers periodic and non-periodic concept drift scenarios, and improves the model's drift detection and response capabilities in complex environments.

[0008] The technical solutions of the present invention are as follows:

[0009] A concept drift detection method based on adaptive data driving includes the following steps:

[0010] Step 1: Preprocess the water treatment plant data and set the prototype neural network parameters;

[0011] Step 2: Adjust data distribution: Extract data distribution characteristics from historical data and combine them with resampling probability to generate the initial data set;

[0012] Step 3: Data mutation, introducing three mutation factors to generate a re-emergence data set;

[0013] Step 4: For the re-emergence dataset, obtain meta-features;

[0014] Step 5: Divide the re-emergence dataset into a support set and a query set and input them into the prototype neural network for pre-training to obtain the trained prototype neural network;

[0015] Step 6: Preprocess the real-time water purification plant data and input it into the trained prototype neural network for concept drift detection.

[0016] According to a preferred embodiment of the present invention, the water purification plant data is preprocessed, including:

[0017] (1) Data conversion: All symbolic data in the water treatment plant data are converted into numerical data, and the input values ​​of the data column are converted into binary form, where "0" represents a normal record and "1" represents an abnormal record;

[0018] (2) Data deduplication: Delete duplicate records in water treatment plant data;

[0019] (3) Data normalization: normalize the water purification plant data after deduplication;

[0020] (4) Divide the data set: Randomly extract a number of data from the water treatment plant data after data normalization as the training set, and the remaining data as the test set.

[0021] Preferably, according to the present invention, extracting data distribution characteristics from historical data and generating a preliminary data set in combination with resampling probability includes:

[0022] The historical data is divided into a set of tasks according to time intervals using formulas (I) and (II). Each task is to predict the distribution of future data based on historical data.

[0023] (I);

[0024] Denotes the tth task, It is expressed as the total training task;

[0025] (II);

[0026] Represented as historical data in the t-th task, Represented as future data in the t-th task;

[0027] Through the prediction model on historical data The feature extractor S captures the data distribution characteristics in the historical data. , x represents the stream data, y represents the target sequence corresponding to x;

[0028] And transmit it to the prediction model, the prediction model uses the data distribution characteristics in the historical data Predict the initial data set. The specific process includes:

[0029] According to the data distribution characteristics in historical data through the prediction model Generate resampling probabilities , which is expressed as formula (III):

[0030] (III);

[0031] Combine the resampling probability with historical data to generate the initial data set , the initial data distribution is calculated by formula (IV):

[0032] (IV);

[0033] Refers to the data distribution of the initial data set.

[0034] Further preferably, the loss function of the prediction model is Use formula (V) to express:

[0035] (V);

[0036] in, represents the Kullback-Leibler divergence, It refers to the data distribution of future datasets;

[0037] The optimization purpose of the prediction model is expressed using formula (VI):

[0038] (VI);

[0039] Represents the optimization target value of the prediction model, It is expressed as the total training task;

[0040] For each iteration of the prediction model, the output value of the prediction model at the nth iteration is determined by formula (VII):

[0041] (VII);

[0042] It is represented as n iterations in the tth task, Parameters set for the prediction model, It is represented as n-1 iterations in the tth task.

[0043] Preferably, according to the present invention, three mutation factors are introduced to generate a re-emergence data set; including:

[0044] Three variation factors are added to the initial dataset to generate four types of data, including:

[0045] Adding a gradual drift factor to the initial data set, that is, increasing or decreasing the data in the initial data set; this is the gradual drift data; adding a discontinuous drift factor to the initial data set, that is, returning the data in the initial data set to zero; this is the discontinuous drift data; adding a sudden change drift factor to the initial data set, that is, randomly changing the data in the initial data set; this is the sudden change drift data; no processing is performed on the initial data set, that is, normal data; these four types of data are collectively referred to as re-sudden data sets. .

[0046] According to the preferred embodiment of the present invention, for the re-emergence data set, the decision tree model is used to obtain the error rate of each timestamp ts, and at the same time, a window mechanism is introduced, and each window The length is m, that is, there are m samples in each window, and the average error rate of the window is Calculated by formula (VIII):

[0047] (VIII);

[0048] is the error rate of the window, m is the number of samples;

[0049] The difference EX between the average error rates of the two windows is used as a meta-feature through formula (IX):

[0050] (IX);

[0051] is the average error rate between two windows, is the error rate of the ts+1th timestamp.

[0052] According to the preferred embodiment of the present invention, the support set and the query set are divided into: There are many categories in the , select k category samples as the support set S = {( , ) , extract the remaining batches from k categories as query set Q = {( , ) ; , To highlight the samples and corresponding labels of the support set in the dataset, To support the number of samples in the set; , To highlight the samples and corresponding labels of the query set in the dataset, is the number of samples in the query set;

[0053] Dynamic evolution factor based on data Introducing the prototype neural network to divide the data distribution of the initial data set Distribution with historical data Formalized as a parameter, the parameter is introduced into the update of the prototype neural network. The specific calculation process is as follows:

[0054] (X);

[0055] (XI);

[0056] In formula (X) and formula (XI), Refers to the data distribution of the initial data set; is the historical data distribution; is a dynamic evolution factor based on data; Represented as the prototype at the t-th iteration, is the embedding function at the tth iteration; Represented as the prototype neural network at the t-1th iteration;

[0057] The prototype interactive distillation mechanism is added to the prototype neural network to establish the self-distillation stage and the mutual distillation stage, and then the self-distillation stage and the mutual distillation stage are integrated with the original prototype stage;

[0058] In the self-distillation stage, the support set features are used Prototype Neural Network Calibration is performed to obtain a more representative prototype representation; in the mutual distillation stage, based on the prototype calibrated in the self-distillation stage, the predicted probability of the query sample class is used as the query prototype of the class; in addition, the query prototype interacts with the original prototype.

[0059] Further preferably, at the original prototype stage, based on each prototype neural network The distance between them is used to obtain the classification probability distribution using the softmax function. , given a query sample The calculation process for category k is as follows:

[0060] (XII);

[0061] in, is the scale factor, Represents the features of the query sample obtained through the feature extraction network, For each task support set, we set an arbitrary target label; the loss function in the original prototype stage is expressed as:

[0062] (XIII);

[0063] Where Nq is the number of query samples in each task, is the loss function in the original prototype stage;

[0064] In the self-distillation stage, the support set features are used and prototype neural networks Get support samples The probability of belonging to a certain category k; by measuring the prototype neural network And support set features to predict the self-distillation loss:

[0065] (XIV);

[0066] (XV);

[0067] Among them, M is the number of samples of category k in the support set, and k represents the support set The corresponding label, is the scale factor, represents the features of the support set obtained by the feature extraction network, Z is the number of target classes in the support set under each task, For each task support set, we can set any target label. is the support set classification probability distribution, is the loss function of the self-distillation stage;

[0068] In the mutual distillation phase, query prototypes is generated from the query sample, using the query prototype and support set features Get support sample x i The probability of belonging to a certain category k; In addition, mutual calibration is performed between the original prototype and the query prototype; The calculation method of the initial generation, t-th generation and mutual distillation loss of the query prototype is as follows:

[0069] (XVI);

[0070] (XVII);

[0071] (XVIII);

[0072] (XIX);

[0073] in, Represents the query prototype, is a dynamic evolution factor based on data, N represents the number of samples belonging to class k in the query set Q, is the number of samples of class k in the query set; Represented as the query prototype at the tth iteration; is the embedding function at the t-1th iteration; represents the features obtained by the feature extraction network of the support set, is the probability distribution of mutual distillation classification, is the loss function of the mutual distillation stage;

[0074] Finally, the total loss results are as follows:

[0075] (XX);

[0076] Among them, L is the total loss of the prototype neural network.

[0077] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned concept drift detection method based on adaptive data driving when executing the computer program.

[0078] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the above-mentioned concept drift detection method based on adaptive data driving.

[0079] A concept drift detection system based on adaptive data driving, comprising:

[0080] The data preprocessing module is configured to: preprocess the water purification plant data and set the prototype neural network parameters;

[0081] The data distribution adjustment module is configured to: extract data distribution features from historical data and generate a preliminary data set based on resampling probability;

[0082] The data mutation module is configured to: introduce three mutation factors to generate a re-emergence dataset;

[0083] The meta-feature obtaining module is configured to: obtain meta-features for the re-emergent dataset;

[0084] The pre-training module is configured to: divide the re-emergence data set into a support set and a query set and input them into the prototype neural network for pre-training to obtain a trained prototype neural network;

[0085] The concept drift detection module is configured to: pre-process the real-time water purification plant data and input it into the trained prototype neural network to perform concept drift detection.

[0086] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0087] Other existing technologies only adapt to the latest historical data and do not proactively predict future data. This method, however, extracts data features from historical data and adjusts the data distribution using resampling probabilities, adapting the model to future data distributions and effectively improving concept detection accuracy. Furthermore, by incorporating a variation factor into the data and performing meta-learning on the model, this method eliminates the need for hypothesis verification to detect concept drift, instead automatically identifying its type and proactively detecting it. These two data distribution adjustments ensure that the model performs well in both unpredictable and predictable future data scenarios.

[0088] By introducing a dynamic evolution factor based on data distribution into prototype generation, the model adjusts the prototype position in real time based on data changes when concepts drift in future data. This allows the model to adapt to changes in data distribution and mitigate biases caused by uneven data distribution. A prototype interactive distillation mechanism is also established to refine the position of prototypes, making them more representative. Incorporating concept drift detection into the water quality testing MLOps data pipeline strengthens MLOps's ability to resist concept drift in future data, reduces the risk of data pipeline anomalies, and improves the stability of water quality data flow within the MLOps data pipeline. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 Schematic diagram of the process for preprocessing water treatment plant data.

[0090] Figure 2 Schematic diagram of adjusted data distribution.

[0091] Figure 3 A schematic diagram of data variation.

[0092] Figure 4 This is the architecture diagram of the prototype neural network.

[0093] Figure 5 Schematic diagram of the concept drift detection method flow. DETAILED DESCRIPTION

[0094] Explanation of terms:

[0095] 1. Meta-learning: It is a machine learning method that aims to enable the model to quickly adapt to new tasks or new environments by learning multiple tasks or multiple data distributions.

[0096] 2. Data distribution: refers to the statistical distribution law of a data set in the feature space, which describes the frequency or probability of data occurrence on different variables (features).

[0097] 3. Pre-training: This means training the model on a large-scale general dataset before learning the main task, so that it can learn common features or representations, and then migrating or fine-tuning them to the target task, thereby improving the model's performance on small samples or complex tasks.

[0098] 4. Kullback-Leibler divergence: It is a method used to measure the difference between two probability distributions.

[0099] 5. Adaptive Hoeffding Tree: This is an incremental decision tree algorithm for data stream classification. It is an improved version of the traditional Hoeffding tree and is designed specifically for processing high-speed, continuously arriving data streams.

[0100] 6. Prediction model FM, existing model: Adaptive Hoeffding Tree.

[0101] 7. Prototype neural network, such as Figure 4 As shown in Figure 1, the basic idea is to create a prototype representation for each category. And for a query that needs to be classified, the distance between the prototype vector of the category and the query point is calculated to determine the distance.

[0102] The operation process of the prototype neural network is as follows:

[0103] 1) First, use the encoding method (embedding function) for each sample to learn the encoding representation of each sample (information extraction).

[0104] 2) After learning the encoding representation for each sample, the encodings of all samples in each category are summed and averaged, and the result is used as the prototype representation of the category. Therefore, the prototype representation of a category is represented by a process of vector summation and averaging. When a new data sample is input into the network, the goal is to predict its classification.

[0105] 3) Use the embedding function for this new data sample to generate its encoded representation.

[0106] 4) Next, the distance between the encoded representation of the new sample and the prototype representation of each category is calculated. The lowest distance is used to determine which category the query sample belongs to. There are no special requirements for distance calculation, and methods such as Euclidean distance and Cosine similarity can be used.

[0107] 5) Finally, after calculating the distances between all the categories, the softmax method is used to convert the distances into probabilities.

[0108] The algorithm is described as follows:

[0109] 1) Assume that the current dataset is D, and the representation of the samples inside it is {(x_1,y_1),(x_2,y_2),....,(x_n,y_n)}, where x represents the vector representation and y represents the classification label.

[0110] 2) For each classification, randomly generate n sample points from the total sample set, and for each classification, generate the final support set S.

[0111] 3) Similarly, randomly select n sample points for each category from the total sample set to generate the query set Q.

[0112] 4) For the sample points within the support set, use the encoding formula fφ (embedding function) to generate a prototype representation for each category. The encoding formula fφ can be any information extraction method, such as CNN, LSTM, etc.

[0113] 5) For each category, generate its prototype

[0114] 6) Similarly, the query set encoding is also generated for the query set.

[0115] 7) Furthermore, we need to calculate the distance between the prototype representations of the query set and the support set.

[0116] 8) Finally, we need to calculate the probability pw(y=k|x) that the current sample belongs to each category. 9) Finally, we calculate the loss function.

[0117] A concept drift detection method based on adaptive data-driven Figure 1 As shown, the following steps are included:

[0118] Step 1: Preprocess the water purification plant data and set the prototype neural network parameters; the prototype neural network parameters include: the number of tasks per training round, the number of categories contained in each task, model structure parameters, distance metric function, embedding network type, and embedding vector dimension;

[0119] Step 2: Adjust data distribution: Extract data distribution characteristics from historical data and combine them with resampling probability to generate the initial data set;

[0120] Step 3: Data mutation, introducing three mutation factors to generate a re-emergence data set;

[0121] Step 4: For the re-emergence dataset, obtain meta-features;

[0122] Step 5: Divide the re-emergence dataset into a support set and a query set and input them into the prototype neural network for pre-training to obtain the trained prototype neural network;

[0123] Step 6: Preprocess the real-time water purification plant data and input it into the trained prototype neural network for concept drift detection. If concept drift occurs in the data, the model will notify the staff. If there is no concept drift, the water quality testing process will continue.

[0124] Example 2

[0125] The difference between the adaptive data-driven concept drift detection method described in Example 1 and the method is as follows:

[0126] Pre-process the water purification plant data (such as power water pressure low 01; power water pressure low 02; power water pressure low 03; compound sodium bisulfate dosing reminder; compound sodium bisulfate lack alarm; compound chlorate dosing reminder; compound chlorate lack alarm; generator is running; compound dosing 1; manual dosing 01; flow dosing 02; compound dosing 03, etc.); Figure 1 Shown, including:

[0127] (1) Data conversion: All symbolic data in the water treatment plant data are converted into numerical data, and the input values ​​of the data column (such as the switch of compound chlorate dosing, "0" represents normal use and "1" represents abnormal use) are converted into binary form, where "0" represents normal record and "1" represents abnormal record; in order to unify the feature expression form.

[0128] (2) Data deduplication: Delete duplicate records in the water treatment plant data; to avoid the classifier being biased towards frequently occurring records during training, and improve its ability to learn less common records.

[0129] (3) Data normalization: Normalize the water treatment plant data after deduplication to eliminate bias in features with large values ​​in the dataset. Data balance is achieved by converting or scaling each feature value to a certain scale range.

[0130] (4) Dataset division: Randomly select 60% of the water treatment plant data after data normalization as the training set, and the remaining 40% as the test set. Ensure a balanced distribution of training and test samples.

[0131] Extract the data distribution characteristics from the historical data and combine them with the resampling probability to generate the initial data set; Figure 2 Shown, including:

[0132] Formula (I) and Formula (II) divide historical data into a set of tasks based on time intervals. Historical data refers to the water purification plant data before the current training time. Each task predicts the future data distribution based on historical data.

[0133] (I);

[0134] Denotes the tth task, It is expressed as the total training task;

[0135] (II);

[0136] Represented as historical data in the t-th task, Represented as future data in the t-th task;

[0137] Through the prediction model FM on historical data The feature extractor S captures the data distribution characteristics in the historical data. , x represents the stream data, y represents the target sequence corresponding to x; the specific implementation process is: by dynamically maintaining an incrementally updated feature space to capture the distribution characteristics of historical data, based on the sliding window online aggregation statistics (such as mean, variance) to reflect the data evolution trend; using LSTM to generate discriminative representation and capture nonlinearity and dependency; finally obtain .

[0138] And transmit it to the prediction model, the prediction model uses the data distribution characteristics in the historical data Predict the initial data set. The specific process includes:

[0139] According to the data distribution characteristics in historical data through the prediction model Generate resampling probabilities , which is expressed as formula (III):

[0140] (III);

[0141] Combine the resampling probability with historical data to generate the initial data set , the initial data distribution is calculated by formula (IV):

[0142] (IV);

[0143] Refers to the data distribution of the initial data set.

[0144] By comparing the distribution of the initial data set ( ) and future data distribution ( The purpose of optimizing the prediction model is to minimize the difference between the initial data distribution and the future data distribution.

[0145] Loss function of the prediction model Use formula (V) to express:

[0146] (V);

[0147] in, represents the Kullback-Leibler divergence, It refers to the data distribution of future datasets;

[0148] The optimization purpose of the prediction model is expressed using formula (VI):

[0149] (VI);

[0150] Represents the optimization target value of the prediction model, It is expressed as the total training task;

[0151] For each iteration of the prediction model, the output value of the prediction model at the nth iteration is determined by formula (VII):

[0152] (VII);

[0153] It is represented as n iterations in the tth task, Parameters set for the prediction model, It is represented as n-1 iterations in the tth task.

[0154] Through the above steps, the data distribution characteristics in the historical data are extracted and combined with the resampling probability to generate the initial data set.

[0155] like Figure 3 As shown in Figure 2, three mutation factors were introduced to generate a re-emergence dataset, including:

[0156] Three variation factors are added to the initial dataset to generate four types of data, including:

[0157] Adding a gradual drift factor to the initial data set, that is, increasing or decreasing the data in the initial data set; this is the gradual drift data; adding a discontinuous drift factor to the initial data set, that is, returning the data in the initial data set to zero; this is the discontinuous drift data; adding a sudden change drift factor to the initial data set, that is, randomly changing the data in the initial data set; this is the sudden change drift data; no processing is performed on the initial data set, that is, normal data; these four types of data are collectively referred to as re-sudden data sets. .

[0158] For the re-emergence dataset, the decision tree model is used to obtain the error rate of each timestamp ts. At the same time, the window mechanism is introduced, and each window The length is m, that is, there are m samples in each window, and the average error rate of the window is Calculated by formula (VIII):

[0159] (VIII);

[0160] is the error rate of the window, m is the number of samples;

[0161] The difference EX between the average error rates of the two windows is used as a meta-feature through formula (IX):

[0162] (IX);

[0163] is the average error rate between two windows, is the error rate of the ts+1th timestamp.

[0164] Prototype neural network is a few-sample learning method based on metric learning. Its core idea is to classify by calculating the distance between the sample and the category prototype. It is suitable for small sample classification tasks. For prototype neural network, it is necessary to divide the support set and query set: There are many categories in the , select k category samples as the support set S = {( , ) , extract the remaining batches from k categories as query set Q = {( , ) ; , To highlight the samples and corresponding labels of the support set in the dataset, To support the number of samples in the set; , To highlight the samples and corresponding labels of the query set in the dataset, is the number of samples in the query set;

[0165] A prototype neural network is used to pre-train four types of data; the four types of data (normal data, gradual drift data, intermittent drift data, and sudden drift data) are defined as four categories, and the prototypes of these four data categories are calculated;

[0166] ;

[0167] in, Represented as prototypes of various categories, The parameters are The embedding function of To support concentrated sample features, Indicates the number of samples in the support set;

[0168] Prototype neural networks generally process static data. For streaming data, static prototypes cannot meet the requirements. In this step, the dynamic evolution factor based on the data is used. Introducing the prototype neural network to divide the data distribution of the initial data set Distribution with historical data Formalized as a parameter, the parameter is introduced into the update of the prototype neural network. The specific calculation process is as follows:

[0169] (X);

[0170] (XI);

[0171] In formula (X) and formula (XI), Refers to the data distribution of the initial data set; is the historical data distribution; is a dynamic evolution factor based on data; Represented as the prototype at the t-th iteration, is the embedding function at the tth iteration; Represented as the prototype neural network at the t-1th iteration;

[0172] The prototype interactive distillation mechanism is added to the prototype neural network to establish the self-distillation stage and the mutual distillation stage, and then the self-distillation stage and the mutual distillation stage are integrated with the original prototype stage;

[0173] In the self-distillation stage, the support set features are used Prototype Neural Network Calibration is performed to obtain a more representative prototype representation. In the mutual distillation stage, the predicted probability of the query sample class is used as the query prototype for that class based on the prototype calibrated in the self-distillation stage. In addition, the query prototype interacts with the original prototype to achieve a mutually reinforcing effect. These two refinement stages are combined into a loss function in the form of constraints.

[0174] In the original prototype stage, based on each prototype neural network The distance between them is used to obtain the classification probability distribution using the softmax function. , given a query sample The calculation process for category k is as follows:

[0175] (XII);

[0176] in, is the scale factor, Represents the features of the query sample obtained through the feature extraction network, For each task support set, we set an arbitrary target label; the loss function in the original prototype stage is expressed as:

[0177] (XIII);

[0178] Where Nq is the number of query samples in each task, is the loss function in the original prototype stage;

[0179] In the self-distillation stage, the support set features are used and prototype neural networks Get support samples The probability of belonging to a certain category k; by measuring the prototype neural network And support set features to predict the self-distillation loss:

[0180] (XIV);

[0181] (XV);

[0182] Among them, M is the number of samples of category k in the support set, and k represents the support set The corresponding label, is the scale factor, represents the features of the support set obtained by the feature extraction network, Z is the number of target classes in the support set under each task, For each task support set, we can set any target label. is the support set classification probability distribution, is the loss function of the self-distillation stage;

[0183] In the mutual distillation phase, query prototypes is generated from the query sample, using the query prototype and support set features Get support sample x i The probability of belonging to a certain category k; In addition, mutual calibration is performed between the original prototype and the query prototype; The calculation method of the initial generation, t-th generation and mutual distillation loss of the query prototype is as follows:

[0184] (XVI);

[0185] (XVII);

[0186] (XVIII);

[0187] (XIX);

[0188] in, Represents the query prototype, is a dynamic evolution factor based on data, N represents the number of samples belonging to class k in the query set Q, is the number of samples of class k in the query set; Represented as the query prototype at the tth iteration; is the embedding function at the t-1th iteration; represents the features obtained by the feature extraction network of the support set, is the probability distribution of mutual distillation classification, is the loss function of the mutual distillation stage;

[0189] Finally, the total loss results are as follows:

[0190] (XX);

[0191] Among them, L is the total loss of the prototype neural network.

[0192] The prototype neural network trained in the above steps is then applied to a real-world scenario. First, real-time water treatment plant streaming data is connected via Kafka. This data is preprocessed to make it acceptable to the model. The trained prototype neural network then performs real-time concept drift detection. If the prototype neural network classifies the data as normal, the data is passed to the next stage for water quality testing. If it is classified as one of the three abnormal data types, the prototype neural network alerts water treatment plant personnel to inspect and repair the water treatment plant equipment.

[0193] The water quality detection process is implemented through a model, which performs water quality detection by judging whether there are abnormal data in the water quality data.

[0194] Table 1 uses data from the water treatment plant from June to December 2024 as input and averages the results over 50 iterations. SFQP is the abbreviation for the algorithm proposed in this invention. True Positive Rate (TPR), also known as Sensitivity or Recall, is defined as the proportion of positive samples correctly identified by the classifier to all true positive samples. Precision (Pre) measures the accuracy of a model or system in identifying positive samples. Accuracy (Acc) measures the consistency between the model's predictions and the true labels. F-Score is a comprehensive metric used to measure a model's performance on positive samples, taking into account both the model's precision (the proportion of samples predicted as positive that are actually positive) and the true positive rate (the proportion of samples predicted as positive that are actually positive). Time represents the algorithm's runtime in seconds.

[0195] Table 1 Result data;

[0196]

[0197] As can be seen from Table 1, SFQP has better performance.

[0198] Example 3

[0199] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the adaptive data-driven concept drift detection method described in embodiment 1 or 2 are implemented.

[0200] Example 4

[0201] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the adaptive data-driven concept drift detection method described in embodiment 1 or 2.

[0202] Example 5

[0203] A concept drift detection system based on adaptive data driving, comprising:

[0204] The data preprocessing module is configured to: preprocess the water purification plant data and set the prototype neural network parameters;

[0205] The data distribution adjustment module is configured to: extract data distribution features from historical data and generate a preliminary data set based on resampling probability;

[0206] The data mutation module is configured to: introduce three mutation factors to generate a re-emergence dataset;

[0207] The meta-feature obtaining module is configured to: obtain meta-features for the re-emergent dataset;

[0208] The pre-training module is configured to: divide the re-emergence data set into a support set and a query set and input them into the prototype neural network for pre-training to obtain a trained prototype neural network;

[0209] The concept drift detection module is configured to: pre-process the real-time water purification plant data and input it into the trained prototype neural network to perform concept drift detection.

[0210] Obviously, the examples listed in the detailed description are only some examples of the present invention, not all examples. Based on the examples of the present invention, all other examples obtained by technicians without creative work should fall within the scope of protection of the present invention.

Claims

1. A concept drift detection method based on adaptive data driving, characterized in that: The following steps are involved: Step 1: Preprocess the water treatment plant data and set the prototype neural network parameters; Step 2: Adjust data distribution: Extract data distribution characteristics from historical data and combine them with resampling probability to generate the initial data set; Step 3: Data mutation, introducing three mutation factors to generate a re-emergence data set; Step 4: For the re-emergence dataset, obtain meta-features; Step 5: Divide the re-emergence dataset into a support set and a query set and input them into the prototype neural network for pre-training to obtain the trained prototype neural network; Step 6: Preprocess the real-time water treatment plant data and input it into the trained prototype neural network for concept drift detection; Preprocessing of water treatment plant data; include: (1) Data conversion: All symbolic data in the water treatment plant data are converted into numerical data, and the input values ​​of the data columns are converted into binary form, where "0" represents a normal record and "1" represents an abnormal record; (2) Data deduplication: deleting duplicate records in water treatment plant data; (3) Data normalization: normalize the water treatment plant data after deduplication; (4) Data set division: randomly select a number of data from the water treatment plant data after data normalization as the training set, and the remaining data as the test set; Extract the data distribution characteristics from the historical data and generate the initial data set based on the resampling probability; including: Through formula (I) and formula (II), historical data is divided into a set of tasks according to time intervals. Each task is to predict the future data distribution based on historical data. Task train ={task (1) ,task (2) ,…,task (t) }(I); task (t) Represented as the tth task, Task train It is expressed as the total training task; Represented as historical data in the t-th task, Represented as future data in the t-th task; Through the prediction model on historical data The feature extractor S captures the data distribution characteristics in the historical data. x represents the stream data, and y represents the target sequence corresponding to x; And transmit it to the prediction model, the prediction model uses the data distribution characteristics in the historical data Predict the initial data set. The specific process includes: According to the data distribution characteristics in historical data through the prediction model Generate resampling probabilities Formula (III): Combine the resampling probability with historical data to generate the initial data set The initial data distribution is calculated using formula (IV): Refers to the data distribution of the initial data set.

2. The adaptive data-driven concept drift detection method according to claim 1, characterized in that: The loss function L of the prediction model FM (task (t) ) is expressed using formula (V): Among them, D KL represents the Kullback-Leibler divergence, It refers to the data distribution of future datasets; The optimization purpose of the prediction model is expressed using formula (VI): FM * Indicates the optimization target value of the prediction model, Task train It is expressed as the total training task; For each iteration of the prediction model, the output value of the prediction model at the nth iteration is determined by formula (VII): FM n+1 It is represented as the n-th iteration process in the t-th task, α is the parameter set by the prediction model, FM n It is represented as n-1 iterations in the tth task.

3. The adaptive data-driven concept drift detection method according to claim 1, characterized in that: Three mutation factors were introduced to generate the re-emergence dataset; they include: Three variation factors are added to the initial dataset to generate four types of data, including: Adding a gradual drift factor to the initial data set, that is, increasing or decreasing the data in the initial data set, is the gradual drift data; adding a discontinuous drift factor to the initial data set, that is, returning the data in the initial data set to zero, is the discontinuous drift data; adding a sudden change drift factor to the initial data set, that is, randomly changing the data in the initial data set, is the sudden change drift data; no processing is performed on the initial data set, that is, normal data; these four types of data are collectively referred to as the re-sudden data set D resam .

4. The adaptive data-driven concept drift detection method according to claim 1, characterized in that: For the re-emergence dataset, the decision tree model is used to obtain the error rate of each timestamp ts. At the same time, the window mechanism is introduced, and each window w i The length is m, that is, there are m samples in each window, and the average error rate of the window is Calculated by formula (VIII): e ts is the error rate of the window, m is the number of samples; The difference between the average error rates of the two windows, EX, is used as a meta-feature through formula (IX): EX i is the average error rate between two windows, is the error rate of the ts+1th timestamp.

5. The adaptive data-driven concept drift detection method according to claim 1, characterized in that: Divide the support set and query set: Re-emerge the dataset D resam There are many categories in the dataset, and k category samples are selected as the support set. Extract the remaining batches from k categories as query sets is the sample and corresponding label of the support set in the re-emergence dataset, n k To support the number of samples in the set; is the sample and corresponding label of the query set in the dataset, n Q is the number of samples in the query set; The data-based dynamic evolution factor σ is introduced into the prototype neural network to divide the data distribution of the initial data set into Distribution with historical data Formalized as a parameter, the parameter is introduced into the update of the prototype neural network. The specific calculation process is as follows: In formula (X) and formula (XI), Refers to the data distribution of the initial data set; is the historical data distribution; σ is the dynamic evolution factor based on data; Represented as the prototype at the t-th iteration, is the embedding function at the tth iteration; Represented as the prototype neural network at the t-1th iteration; The prototype interactive distillation mechanism is added to the prototype neural network to establish the self-distillation stage and the mutual distillation stage, and then the self-distillation stage and the mutual distillation stage are integrated with the original prototype stage; In the self-distillation stage, the support set features are used Prototype neural network c k Calibration is performed to obtain a more representative prototype representation; in the mutual distillation stage, based on the prototype calibrated in the self-distillation stage, the predicted probability of the query sample class is used as the query prototype of the class; in addition, the query prototype interacts with the original prototype.

6. The adaptive data-driven concept drift detection method according to any one of claims 1 to 5, characterized in that: In the original prototype stage, based on each prototype neural network c k The distance between them is used to obtain the classification probability distribution using the softmax function. Given a query sample x q The calculation process for category k is as follows: Where β is the scaling factor, represents the features obtained by the feature extraction network for the query sample, z, is an arbitrary target label in the support set for each task; the loss function in the original prototype stage is expressed as: Where Nq is the number of query samples in each task, L o is the loss function in the original prototype stage; In the self-distillation stage, the support set features are used and prototype neural network c k Get support sample x i The probability of belonging to a certain category k; by measuring the prototype neural network c k And support set features to predict the self-distillation loss: Among them, M is the number of samples of category k in the support set, and k represents the support set x i The corresponding label, β is the scaling factor, represents the features obtained by the feature extraction network in the support set, Z is the number of target classes in the support set under each task, and z, is any target label in the support set of each task. is the support set classification probability distribution, L s is the loss function of the self-distillation stage; In the mutual distillation phase, query prototypes is generated from the query sample, using the query prototype and support set features Get support sample x i The probability of belonging to a certain category k; In addition, mutual calibration is performed between the original prototype and the query prototype; The calculation method of the initial generation, t-th generation and mutual distillation loss of the query prototype is as follows: in, represents the query prototype, σ is the dynamic evolution factor based on data, N represents the number of samples belonging to class k in the query set Q, is the number of samples of class k in the query set; Represented as the query prototype at the tth iteration; is the embedding function at the t-1th iteration; represents the features obtained by the feature extraction network of the support set, is the probability distribution of mutual distillation classification, L i is the loss function of the mutual distillation stage; Finally, the total loss results are as follows: L=L o +L s +L i (XX); Among them, L is the total loss of the prototype neural network.

7. A concept drift detection system based on adaptive data driving, used to implement the concept drift detection method based on adaptive data driving according to any one of claims 1 to 6, characterized in that: include: The data preprocessing module is configured to: preprocess the water purification plant data and set the prototype neural network parameters; The data distribution adjustment module is configured to: extract data distribution features from historical data and generate a preliminary data set based on resampling probability; The data mutation module is configured to: introduce three mutation factors to generate a re-emergence dataset; The meta-feature obtaining module is configured to: obtain meta-features for the re-emergent dataset; The pre-training module is configured to: divide the re-emergence data set into a support set and a query set and input them into the prototype neural network for pre-training to obtain a trained prototype neural network; The concept drift detection module is configured to: pre-process the real-time water purification plant data and input it into the trained prototype neural network to perform concept drift detection.

Images