Metalearning anomaly detection method and system under data full life cycle

Abstract

The invention discloses a meta-learning anomaly detection method and system under a data full life cycle, and the method comprises the steps: collecting an original data flow according to a preset sampling frequency, and obtaining an original data set with a stage label; null value-zero value cleaning is carried out on the data set, and a coarse screening data set is output; dividing into a plurality of meta-learning tasks according to life cycle stages, adopting a gradient-based model-independent meta-learning framework, carrying out external loop and internal loop iteration by taking the tasks as units, obtaining stage-independent initialization parameters, and generating a stage self-adaptive anomaly detection model; and calculating an abnormal score for the unlabeled query set and the to-be-measured stream data, performing rechecking in combination with a median deviation threshold, and outputting a final abnormal data set. According to the invention, through meta-learning task division in a full life cycle stage and a gradient-based model-independent meta-learning framework, the problems of significant data feature difference in different stages of a data full life cycle, insufficient generalization ability of an anomaly detection model and difficulty in adapting to inter-stage concept drift are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and anomaly detection, and in particular to a meta-learning anomaly detection method and system for data throughout its life cycle. Background Art

[0002] With the rapid development of big data, artificial intelligence, and information system technologies, anomaly detection at various stages of data lifecycle has gradually become an important means to ensure data quality and stable system operation. In recent years, meta-learning, as a machine learning method that can achieve efficient learning with small samples and quickly adapt to new tasks, has received extensive attention and research in the field of anomaly detection.

[0003] Existing meta-learning anomaly detection methods mostly use static or fixed model parameters. In actual production environments, when the system operating environment or data features experience concept drift, the model's detection effect will be significantly reduced, making it difficult to maintain high detection accuracy over the long term. At the same time, there is currently a lack of a unified stage-adaptive technical framework for anomaly detection in multi-stage data lifecycle scenarios. It is usually necessary to design a separate model for each stage, resulting in high development costs, slow response speed, and the inability to effectively share existing knowledge and parameters between stages.

[0004] CN114239712A discloses an anomaly detection method based on a meta-learning architecture of heterogeneous information networks. This method captures the structural information, heterogeneous features and unlabeled information of heterogeneous information networks through graph neural networks, thereby achieving effective migration between different networks. CN119939445A discloses a few-sample anomaly detection method based on meta-learning and a multimodal large model. This method constructs a semantic-anomaly network through visual and language encoders to enhance the generalization ability and robustness of cross-category anomaly detection and improve the accuracy of anomaly detection. Although the above method has achieved remarkable results in specific fields, it mainly focuses on anomaly detection in static data scenarios or fixed category tasks, and does not fully consider the differential characteristics of data at various stages of the entire life cycle and its dynamic changes. Therefore, it is difficult to cope with the rapid adaptation problem when there are significant differences in data characteristics at different stages of the data life cycle, which limits its application effect and generalization performance in complex information system scenarios.

[0005] Therefore, there is an urgent need to develop an anomaly detection technology solution that can take into account the differences in data features at all stages of the entire life cycle and can quickly and adaptively adjust when data features drift, so as to meet the urgent needs for real-time, robust, and high-precision anomaly detection in complex information systems.

[0006] In summary, existing anomaly detection technologies have problems such as poor stage generalization ability, untimely response to concept drift, and difficulty in effectively reusing knowledge in life cycle stages. We have invented a meta-learning anomaly detection method for the entire data life cycle. Through the division of meta-learning tasks in the entire life cycle stages, a gradient-based model-independent meta-learning framework, and a stage-adaptive online update mechanism, we effectively solve the problems of significant differences in data features at different stages of the data life cycle, insufficient generalization ability of anomaly detection models, and difficulty in adapting to concept drift between stages. Summary of the Invention

[0007] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract of the specification and the title of the invention of this application to avoid blurring the purpose of this section, the abstract of the specification and the title of the invention, and such simplifications or omissions cannot be used to limit the scope of the invention.

[0008] In view of the above existing problems, the present invention is proposed.

[0009] To solve the above technical problems, the present invention provides the following technical solution: According to the preset sampling frequency, the original data stream of the target information system is collected in the six stages of the data life cycle: generation, transmission, storage, processing, archiving and destruction, and the original data set D0 with stage labels is obtained;

[0010] The original data set D0 is cleaned of null values ​​and zero values, and cross-limit filtering is performed according to the adaptive upper and lower limit dynamic thresholds and interquartile range criteria of each life cycle stage to output a coarse screening data set D1;

[0011] The coarse screening dataset D1 is divided into several meta-learning tasks T according to the life cycle stage. i , extract the feature vector of each task and label the support sample set S i and the unlabeled query set Q i ;

[0012] A gradient-based model-independent meta-learning framework is used to i Iterate the outer and inner loops for the unit to obtain the stage-independent initialization parameter θ * ;

[0013] For the target phase data stream collected in real time, the initialization parameter θ is used * And the labeled support sample set S i Perform K-step gradient updates to generate a phase-adaptive anomaly detection model;

[0014] The anomaly detection model is used to detect the unlabeled query set Q iThe anomaly score is calculated based on the tested stream data, and then reviewed with the median deviation threshold Z1 to output the final anomaly dataset A.

[0015] When it is detected that the stage distribution drift metric exceeds the preset threshold Z2, the incremental meta-learning process is triggered to retrain.

[0016] As a preferred solution of the meta-learning anomaly detection method for the entire data life cycle of the present invention, the original data set D0 with stage labels is obtained, including:

[0017] Deploy data collection agents at the six stages of generation, transmission, storage, processing, archiving and destruction of the target information system;

[0018] Synchronously trigger sampling at each data collection agent point based on a unified sampling time benchmark, and write a unique timestamp for each sampling batch;

[0019] Write the sampled raw data into the temporary buffer, and add the stage label, data source node ID, sampling frequency and sampling accuracy;

[0020] After passing the buffer consistency check, the appended original data is written into the original data set D0 in batches, and the write batch number is recorded;

[0021] The original data set D0 at least includes data timestamp, data value, data source node ID, corresponding life cycle stage label, sampling frequency and sampling accuracy information.

[0022] As a preferred solution of the meta-learning anomaly detection method for the entire data life cycle of the present invention, the output coarse screening dataset D1 includes:

[0023] Perform a field integrity check on the original data set D0 and remove data rows with missing key fields;

[0024] Fill missing markers for fields detected as all zeros or empty strings and move them into the abnormal candidate table;

[0025] According to the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion;

[0026] The data samples that do not exceed the limit and have complete fields are written into the coarse screening data set D1, and the samples that exceed the limit are moved into the abnormal candidate table.

[0027] As a preferred solution of the meta-learning anomaly detection method under the whole life cycle of data described in the present invention, the coarse screening dataset D1 is divided into several meta-learning tasks T according to the life cycle stage. i ,include:

[0028] In each life cycle stage, a sliding window with a fixed length of r is set and slides with a step size of r1. The data in the window and its stage label form a candidate sub-dataset;

[0029] The candidate sub-dataset is processed according to the data volume threshold N min Filter and retain the sub-datasets that meet the data volume requirements as a task T i ;

[0030] For each task T i Writes unique task number, stage label, and window time range metadata.

[0031] As a preferred solution of the meta-learning anomaly detection method under the data life cycle of the present invention, the feature vector of each task extraction stage and the annotation support sample set S i and the unlabeled query set Q i ,include:

[0032] The stage feature vector is composed of statistical features, frequency domain features and time series features, wherein the statistical features include mean, variance, skewness and kurtosis, the frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum, and the time series features include autocorrelation coefficient and volatility;

[0033] The labeled support sample set S i Select by stratified sampling to obtain normal samples and suspected abnormal samples, and the total number of samples |Si|≦m;

[0034] The unlabeled query set Q i The remaining data of the task window are arranged in time sequence to ensure Q i With S i There is no overlap and the full window period is covered.

[0035] As a preferred solution of the meta-learning anomaly detection method under the data life cycle of the present invention, the initialization parameter θ * ,include:

[0036] Randomly select B tasks T from each life cycle stage in turn i Form a group of training tasks;

[0037] For each task T in the training task i , in its annotation support sample set S i Perform n steps of gradient update on the model to obtain the temporary model parameters θ i , and calculate the temporary model parameters θ i In the corresponding unlabeled query set Q i The loss value L on i ;

[0038] The loss value L for all tasks i Find the average, perform a one-element gradient update on the original model parameters θ according to the outer loop learning rate η1 to obtain the new model parameters;

[0039] Repeat the above steps until the maximum number of iterations M is reached;

[0040] Solidify the model parameters after the final convergence into the stage-independent initialization parameters θ * .

[0041] As a preferred solution of the meta-learning anomaly detection method for the entire data lifecycle of the present invention, the adaptive anomaly detection model in the generation phase includes:

[0042] For the target stage data stream collected in real time, a new supporting sample buffer S′ of this stage is constructed;

[0043] Initialize the parameter θ * As the starting point, only the weights of the density estimation module are unfrozen, and n1 steps of mini-batch gradient update are performed on the new support sample buffer S′. The inner loop learning rate is set to η0, and η0 decreases exponentially with the batch;

[0044] After each update, immediately evaluate the loss on the micro validation set V′ collected at the same time as the new support sample buffer S′. If the loss does not decrease for q consecutive times, stop updating;

[0045] The model parameters obtained when the update stops are recorded as θ′, and the corresponding model is the stage-adaptive anomaly detection model.

[0046] As a preferred solution of the meta-learning anomaly detection method for the entire data life cycle of the present invention, the output of the final anomaly dataset A includes:

[0047] Calculate the anomaly score for each sample to be tested;

[0048] Then the initial screening abnormality is determined based on the median deviation threshold Z1;

[0049] Perform a stage consistency check on the samples with initial abnormalities, and confirm them as final abnormalities when the abnormality score exceeds the threshold Z2;

[0050] For samples that are confirmed to be abnormal, their abnormal category, occurrence stage and timestamp are recorded and stored in the abnormal dataset A.

[0051] As a preferred solution of the meta-learning anomaly detection system for data throughout the life cycle of the present invention, the system includes: one or more processors;

[0052] A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations, including the process of the aforementioned meta-learning anomaly detection method under the entire data life cycle.

[0053] As a preferred embodiment of a computer-readable medium storing software according to the present invention, the software includes instructions that can be executed by one or more computers, and the instructions, through such execution, enable the one or more computers to perform operations, and the operations include the process of the meta-learning anomaly detection method under the entire life cycle of data as described above.

[0054] Beneficial effects of the present invention:

[0055] 1. By obtaining raw data sets with stage labels, we achieve accurate and systematic collection of data from each key stage of the target information system, thus providing a comprehensive and detailed data foundation for anomaly detection, effectively covering the entire data lifecycle. This solves the problem of traditional anomaly detection methods that only undersample at specific stages or under static data conditions, improves the integrity and accuracy of the data acquisition process, and enhances the reliability of anomaly detection from the source.

[0056] 2. By obtaining a coarse-screened data set, we can effectively remove obvious anomalies and interference information from the original data, improve the accuracy and stability of subsequent analysis, and avoid false positives in the anomaly detection model due to noisy data. This results in a high degree of data purification and stable data quality.

[0057] 3. By designing differentiated feature representations and task learning strategies based on the characteristics of data at different stages, we overcome the problem that traditional methods cannot fully express the heterogeneity of data at different stages, achieving the effect of clear tasks and accurate representation of data features;

[0058] 4. By obtaining stage-independent initialization parameters, the extraction and transfer of cross-stage anomaly detection knowledge is achieved, enabling rapid generalization and adaptation to new tasks at any stage in the lifecycle. This addresses the poor generalization performance of traditional anomaly detection models, significantly improves the model's rapid adaptability and cross-stage generalization capabilities, reduces the cost of model training in new stages, and improves deployment efficiency.

[0059] 5. By generating an adaptive anomaly detection model during the generation phase, rapid online fine-tuning of the anomaly detection model is achieved, thus avoiding the problem of reduced accuracy of the model due to phase changes or concept drift. This achieves the effect of real-time response to data changes and efficient and accurate adjustment, ensuring the online performance and reliability of the anomaly detection model.

[0060] 6. By further performing detailed verification and neighborhood check on the initially detected abnormal data, the high false alarm rate problem in existing technologies is overcome, achieving highly reliable anomaly identification results and extremely low false alarm rates.

[0061] 7. By dynamically adjusting model parameters, the problem of gradually declining performance in existing anomaly detection technologies over the long term is effectively overcome, achieving the effect of stable maintenance and active updating of anomaly detection accuracy in long-term use environments, ensuring the long-term and stable application of anomaly detection technology in complex information system environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0063] Figure 1 Schematic diagram of the process of meta-learning anomaly detection method for data throughout its life cycle shown in the present invention. DETAILED DESCRIPTION

[0064] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, but not all of the embodiments.

[0065] Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without making any creative work should fall within the scope of protection of the present invention.

[0066] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0067] According to an embodiment of the present invention, Figure 1 The flowchart shown in the figure shows a meta-learning anomaly detection method for the entire data life cycle, which specifically includes the following steps:

[0068] S1. According to the preset sampling frequency, the original data streams of the target information system are collected in the six stages of the data life cycle: generation, transmission, storage, processing, archiving, and destruction, to obtain the original data set D0 with stage labels. Among them, it is necessary to explain the following in this step:

[0069] Deploy data collection agents at the six stages of target information system generation, transmission, storage, processing, archiving and destruction;

[0070] Synchronously trigger sampling at each data collection agent point based on a unified sampling time benchmark, and write a unique timestamp for each sampling batch;

[0071] Write the sampled raw data into the temporary buffer, and add the stage label, data source node ID, sampling frequency and sampling accuracy;

[0072] After passing the buffer consistency check, the appended original data is written into the original data set D0 in batches, and the write batch number is recorded;

[0073] The original data set D0 at least includes data timestamp, data value, data source node ID, corresponding life cycle stage label, sampling frequency and sampling accuracy information.

[0074] As an example, each agent point controls the local system clock drift within ±1ms through a high-precision PTP (Precision Time Protocol) network clock (such as a master-slave clock system based on IEEE 1588-2008 Precision Time Protocol), and triggers sampling based on Coordinated Universal Time (UTC) as a unified time base.

[0075] In an optional embodiment, the preset sampling frequency is configured according to the real-time level of the service, with a default of 1 Hz (once per second) and can be dynamically adjusted within a range of 0.1 Hz to 10 kHz:

[0076] The high real-time stage (generation, transmission) can be increased to ≥100Hz;

[0077] The low-variability stage (archiving, destruction) can be reduced to ≤0.1Hz;

[0078] The frequency configuration is uniformly distributed by the central scheduling module, and the sampling agent automatically takes effect in the next full second after receiving the parameter update.

[0079] In an optional embodiment, at the full second UTC 0ms, each proxy point synchronously triggers a sampling, obtains the original data packet of the stage and immediately writes it into the local temporary buffer. Exemplarily, the temporary buffer adopts a write-delay parallel verification strategy, where:

[0080] Calculate whether the number of records in the batch is consistent with the number of node configurations;

[0081] Verify that the timestamp difference of all entries is ≤1ms;

[0082] Verify that the PhaseTag matches the node role by comparing it with the phase enumeration table;

[0083] Data batches that fail verification are automatically marked as invalid and placed in the exception buffer queue, awaiting operation and maintenance audit;

[0084] The data batches that pass the consistency check are written into the original data set D0 in batches within t+Δt (Δt≤200ms).

[0085] S2: Clean the original dataset D0 of null values ​​and zero values, and perform cross-limit filtering based on the adaptive upper and lower dynamic thresholds and interquartile range criteria for each life cycle stage, and output the coarse screening dataset D1.

[0086] Perform field integrity check on the original dataset D0 and remove data rows with missing key fields;

[0087] Fill missing markers for fields detected as all zeros or empty strings and move them into the exception candidate table;

[0088] According to the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion;

[0089] The data samples that do not exceed the limit and have complete fields are written into the coarse screening data set D1, and the samples that exceed the limit are moved into the abnormal candidate table.

[0090] As an example, extract the corresponding subset according to the life cycle stage s∈{0,…,5} right Calculation stage reference value (The median is taken by default), according to the configuration management coefficient λ s , γ s (0<λ s , γ s <1) Calculate the upper and lower dynamic thresholds:

[0091]

[0092] Among them, U s is the upper threshold of stage s, L s is the lower threshold of stage s;

[0093] For example, by Calculate the lower quartile and the upper quartile get

[0094] Define the limit coefficient κ as 1.5, and judge for each sample value v:

[0095]

[0096] At the same time, determine whether v satisfies L s ≤v≤U s ;

[0097] If any condition of v is not met, the corresponding record is moved to the exception candidate table E_cand and marked as out_of_range.

[0098] It should be noted that a record is written into the coarse screening dataset D1 only if it meets the following conditions at the same time:

[0099] (1) The fields are complete;

[0100] (2) non-null value, non-zero value;

[0101] (3)L s ≤v≤U s and is not an IQR outlier;

[0102] For the records written into the coarse screening dataset D1, the original six-element fields {ts, value, node_id, phase, freq, precision} are retained, and qc_flag = 0 is added to indicate that the quality inspection has passed;

[0103] For the records transferred to the exception candidate table E_cand, the qc_flag=1 and error_tag fields are added to identify the exception type for subsequent review.

[0104] S3, divide the coarse screening dataset D1 into several meta-learning tasks T according to the life cycle stage i , extract the feature vector of each task and label the support sample set S i and the unlabeled query set Q i Among them, the following points need to be explained in this step:

[0105] In each life cycle stage, a sliding window with a fixed length of r is set and slides with a step size of r1. The data in the window and its stage label form a candidate sub-dataset;

[0106] The candidate sub-datasets are classified according to the data volume threshold N min Filter and retain the sub-datasets that meet the data volume requirements as a task T i ;

[0107] For each task T i Writes unique task number, stage label, and window time range metadata.

[0108] In an optional embodiment, for each task, the feature vector of the extraction stage and the annotation support sample set S are extracted. i and the unlabeled query set Q i ,include:

[0109] The stage feature vector is composed of statistical features, frequency domain features and time series features. Statistical features include mean, variance, skewness and kurtosis; frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum; and time series features include autocorrelation coefficient and volatility.

[0110] Annotated support sample set S i Select by stratified sampling to obtain normal samples and suspected abnormal samples, and the total number of samples |Si|≦m;

[0111] Unlabeled query set Q i The remaining data of the task window are arranged in time sequence to ensure Q i With S i There is no overlap and the full window period is covered.

[0112] S4, using a gradient-based model-independent meta-learning framework, with task T i Iterate the outer and inner loops for the unit to obtain the stage-independent initialization parameter θ * Among them, the following points need to be explained in this step:

[0113] Randomly select B tasks T from each life cycle stage in turn i Form a group of training tasks;

[0114] For each task T in the training task i , in its annotation support sample set S i Perform n steps of gradient update on the model to obtain the temporary model parameters θ i , and calculate the temporary model parameters θ i In the corresponding unlabeled query set Q i The loss value L on i ;

[0115] The loss value L for all tasks i Find the average, perform a one-element gradient update on the original model parameters θ according to the outer loop learning rate η1 to obtain the new model parameters;

[0116] Repeat the above steps until the maximum number of iterations M is reached;

[0117] Solidify the model parameters after the final convergence into the stage-independent initialization parameters θ * .

[0118] S5, for the target stage data stream collected in real time, use the initialization parameter θ * And the labeled support sample set S i Perform K-step gradient updates to generate the adaptive anomaly detection model. The following points should be noted in this step:

[0119] When the target phase real-time data stream reaches w consecutive sampling cycles, the system creates a new supporting sample buffer S′ in the memory, where the ratio of normal samples to suspected abnormal samples is not less than 3:1. The upper limit of the buffer size |S′| is dynamically issued by the central scheduling module;

[0120] At the same time, v samples are randomly selected from the same time window to form a validation set V′, and locked until the end of this round of online update;

[0121] Initialize the parameter θ independently of the offline phase * As a starting point, only the weights of the density estimation submodule G are unfrozen, and the rest of the feature embedding submodule E remains frozen to reduce the amount of online fine-tuning calculations;

[0122] Divide S′ into batch size b Small batches, set the inner loop learning rate η0;

[0123] Perform a forward-backward propagation for each mini-batch and update the parameters of G for a total of n1 steps;

[0124] After each update step, the loss value is calculated on the validation set V′. If the loss value does not decrease significantly within q consecutive iterations, the update is stopped.

[0125] The parameter when the update stops is recorded as θ′ and written into the model registry to generate the version number ver=<phase,time> ;

[0126] Its corresponding model f θ′ This is the stage-adaptive anomaly detection model, which can be directly called in step S6.

[0127] As an example, the mathematical expression of the stage-adaptive anomaly detection model is:

[0128]

[0129] Among them, x is the sample vector to be tested, θ′ is the model parameter set after online update, σ is the Logistic compression function, Z is the normalization constant, T is the buffer window length, Ψ α (x, τ) is a nonlinear mapping with time offset, Φ β (·) is the logarithmic gain function of logarithmic magnitude modulation, s i is the eigenvector of the support sample i, κ i is the kernel density weight, η is the adaptive scale of the Gaussian kernel bandwidth, γ is a stabilization constant to prevent the denominator from being zero, d is the feature dimension, and δ is used to suppress the growth of extreme integral values;

[0130] It should be noted that when f θ′ When (x)≤0.5, it is a normal sample; when 0.5 <f θ′When (x)≤0.8, it is a suspicious sample and needs to be further checked with the threshold Z1 in step S6; when f θ′ When (x)>0.8, it is directly marked as a high-confidence anomaly.

[0131] S6. Use anomaly detection model to detect unlabeled query set Q i The anomaly score is calculated based on the flow data to be tested, and the median deviation threshold Z1 is combined for review to output the final anomaly dataset A. The following points need to be explained in this step:

[0132] Call the stage adaptive anomaly detection model f obtained in step S5 θ′ , for the unlabeled query set Q i And the real-time stream data to be tested is input one by one, and the original score p(x)∈(0,1) is output;

[0133] Combine support and confidence interval to construct a comprehensive anomaly score:

[0134]

[0135] in, is the mean of p(x) in the current time window, ξ>0 is the Sigmoid steepness scheduling coefficient, and Score(x) is the comprehensive anomaly score;

[0136] Calculate the comprehensive score set {Score(x j )} median Med and absolute median difference MAD = median(|Score(x j )-Med|);

[0137] Let the median deviation threshold Z1 = Med + β·MAD, β∈[2,4];

[0138] When Score(x)>Z1, mark sample x as an initial screening anomaly;

[0139] For the initially screened abnormal samples, a ring neighbor sequence is established according to their phase label phase. If Score(x)>Z2 (global high confidence threshold, Z2∈(0.85,0.95)), the sample x is confirmed to be the final abnormality;

[0140] Otherwise, it will be downgraded to suspicious and re-inspected in the next window;

[0141] For confirmed abnormal samples, record "timestamp ts, source node node_id, life cycle phase phase, comprehensive score Abnormal level (high / medium), initial screening reason (MAD or high confidence Z2)" generates a unique abnormal ID and writes it into abnormal dataset A;

[0142] The abnormal dataset A is stored in columnar format and partitioned by phase→ts. An incremental backup is triggered every time m records are written, and the newly added abnormality ID and phase are pushed to the monitoring system through the alarm bus to achieve real-time early warning.

[0143] S7: When it is detected that the stage distribution drift metric δ exceeds the preset threshold Z2, the incremental meta-learning process is triggered to retrain. Among them, it should be noted that:

[0144] In each life cycle stage s, set the length to w R The reference window R s , and open the length w M The motion detection window M s ,When the detection window is filled with the latest sample, the stage distribution drift metric is calculated as the maximum mean difference Δs;

[0145] Configure warning thresholds for each stage With mandatory threshold

[0146] like It is a stable state, and only the detection window is scrolled;

[0147] like It is a light drift state, and a quick fine-tuning process is performed;

[0148] like If it is in a serious drift state, perform the incremental retraining process.

[0149] As an example, a quick fine-tuning process is performed in a light drift state, which includes the following steps:

[0150] From the current stage recently w M samples to construct the incremental support set S r With the validation set V r ;

[0151] The current stage adaptive model f θ′ As a starting point, only the last h layers of the density estimation module are unfrozen, using a learning rate η fast Perform small batch updates;

[0152] If V r The upper loss decreases compared to before the update ≥∈ fast That is, accept the new parameters, otherwise roll back and keep the old model;

[0153] Update reference window R s For the most recent R Stable samples; record model version ver fast and resume monitoring.

[0154] As an example, the incremental retraining process is performed under severe drift conditions, which includes the following steps:

[0155] In stage s, with window w R Re-divide the coarse screening data for the unit and generate a temporary task set

[0156] since Randomly select B inc Tasks and historical task library T hist B in hist Steady-state tasks are spliced ​​into incremental training batches inc ;

[0157] The new initialization parameters Test on the independent validation task set of stage s, if the average AUC ≥ AUC min And it is improved by ≥δ compared with the old initialization gain Then replace the global initialization parameters Otherwise, keep the original value and record the alarm;

[0158] After successful replacement, the version number ver is generated inc And distribute to online nodes;

[0159] If the subsequent q fail If a detection window triggers severe drift continuously, it will automatically roll back to the last stable version. prev .

[0160] It should be noted that drift detection and fine-tuning / retraining jobs are triggered by the central orchestrator through an asynchronous event queue. All computing resource consumption, time consumption, and version evolution information are written to the log table. When the memory or GPU usage of the retraining job exceeds the preset threshold, it is dynamically downgraded to a sequential task to ensure the priority of online inference.

[0161] Preferably, the present invention realizes continuous quantitative monitoring and hierarchical response to changes in data distribution in the life cycle stage, quickly fine-tunes mild drift to maintain low latency, and triggers incremental meta-learning for high-intensity drift to ensure long-term robustness of the model. At the same time, it relies on dual thresholds and rollback mechanisms to avoid frequent shocks and ensure the stability of anomaly detection services.

[0162] The above-mentioned feature vector, the extraction method of the labeled support sample set and the unlabeled query set, and the loss value L i The calculation method can be performed using the methods and means in the existing technology, which will not be described in detail in this example.

[0163] In application of the above embodiments, other aspects disclosed in the embodiments of the present invention further propose a meta-learning anomaly detection system for the entire data life cycle, including: one or more processors and a memory.

[0164] The memory is used to store operable instructions, which, when executed by the one or more processors, cause the one or more processors to perform operations, including the process of a meta-learning anomaly detection method under the entire life cycle of data in the aforementioned embodiment, especially Figure 1 The process of the method shown.

[0165] Other aspects disclosed in the embodiments of the present invention further provide a computer-readable medium storing software, wherein the software includes instructions that can be executed by one or more computers, and the execution of these instructions causes one or more computers to perform operations, including the process of a meta-learning anomaly detection method for data throughout the life cycle of the aforementioned embodiment, especially Figure 1 The process of the method shown.

[0166] It should be appreciated that embodiments of the present invention can be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory.

[0167] The method may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes a computer to operate in a specific and predefined manner.

[0168] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system, however, the program can be implemented in assembly or machine language if desired.

[0169] In any case, the language may be a compiled or interpreted language.

[0170] Furthermore, the program can be run on an application specific integrated circuit programmed for this purpose.

[0171] The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively executes on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that can be executed by one or more processors.

[0172] Further, the method may be implemented in any type of computing platform operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device.

[0173] Aspects of the present invention may be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, etc., such that it can be read by a programmable computer and, when the storage medium or device is read by the computer, can be used to configure and operate the computer to perform the processes described herein.

[0174] Additionally, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network.

[0175] The invention described herein includes these and other various types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor.

[0176] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A meta-learning anomaly detection method for data throughout its life cycle, characterized in that: include: According to the preset sampling frequency, the original data streams of the target information system in the six stages of data life cycle, namely generation, transmission, storage, processing, archiving and destruction, are collected to obtain the original data set D0 with stage labels; The original data set D0 is cleaned of null values ​​and zero values, and cross-limit filtering is performed according to the adaptive upper and lower limit dynamic thresholds and interquartile range criteria of each life cycle stage to output a coarse screening data set D1; The coarse screening dataset D1 is divided into several meta-learning tasks T according to the life cycle stage. i , extract the feature vector of each task and label the support sample set S i and the unlabeled query set Q i ; A gradient-based model-independent meta-learning framework is used to i Iterate the outer and inner loops for the unit to obtain the stage-independent initialization parameter θ * ; For the target phase data stream collected in real time, the initialization parameter θ is used * And the labeled support sample set S i Perform K-step gradient updates to generate a phase-adaptive anomaly detection model; The anomaly detection model is used to detect the unlabeled query set Q i The anomaly score is calculated based on the tested stream data, and then reviewed with the median deviation threshold Z1 to output the final anomaly dataset A. When it is detected that the stage distribution drift metric exceeds the preset threshold Z2, the incremental meta-learning process is triggered to retrain.

2. The meta-learning anomaly detection method for data in the entire life cycle according to claim 1 is characterized in that: The original dataset D0 with stage labels is obtained, including: Deploy data collection agents at the six stages of generation, transmission, storage, processing, archiving and destruction of the target information system; Synchronously trigger sampling at each data collection agent point based on a unified sampling time benchmark, and write a unique timestamp for each sampling batch; Write the sampled raw data into the temporary buffer, and add the stage label, data source node ID, sampling frequency and sampling accuracy; After passing the buffer consistency check, the appended original data is written into the original data set D0 in batches, and the write batch number is recorded; The original data set D0 at least includes data timestamp, data value, data source node ID, corresponding life cycle stage label, sampling frequency and sampling accuracy information.

3. The meta-learning anomaly detection method for data in the entire life cycle according to claim 1 or 2, characterized in that: The output coarse screening data set D1 includes: Perform a field integrity check on the original data set D0 and remove data rows with missing key fields; Fill missing markers for fields detected as all zeros or empty strings and move them into the abnormal candidate table; According to the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion; The data samples that do not exceed the limit and have complete fields are written into the coarse screening data set D1, and the samples that exceed the limit are moved into the abnormal candidate table.

4. The meta-learning anomaly detection method for data in the entire life cycle according to claim 3 is characterized in that: The coarse screening dataset D1 is divided into several meta-learning tasks T according to the life cycle stage. i ,include: In each life cycle stage, a sliding window with a fixed length of r is set and slides with a step size of r1. The data in the window and its stage label form a candidate sub-dataset; The candidate sub-dataset is processed according to the data volume threshold N min Filter and retain the sub-datasets that meet the data volume requirements as a task T i ; For each task T i Writes unique task number, stage label, and window time range metadata.

5. The meta-learning anomaly detection method for data in the entire life cycle according to claim 4 is characterized in that: Extract the feature vector and label the support sample set S for each task i and the unlabeled query set Q i ,include: The stage feature vector is composed of statistical features, frequency domain features and time series features, wherein the statistical features include mean, variance, skewness and kurtosis, the frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum, and the time series features include autocorrelation coefficient and volatility; The labeled support sample set S i Select by stratified sampling to obtain normal samples and suspected abnormal samples, and the total number of samples |Si|≦m; The unlabeled query set Q i The remaining data of the task window are arranged in time sequence to ensure Q i With S i There is no overlap and the full window period is covered.

6. The meta-learning anomaly detection method for data throughout its life cycle according to claim 1, characterized in that: The acquisition stage is independent of the initialization parameter θ * ,include: Randomly select B tasks T from each life cycle stage in turn i Form a group of training tasks; For each task T in the training task i , in its annotation support sample set S i Perform n steps of gradient update on the model to obtain the temporary model parameters θ i , and calculate the temporary model parameters θ i In the corresponding unlabeled query set Q i The loss value L on i ; The loss value L for all tasks i Find the average, perform a one-element gradient update on the original model parameters θ according to the outer loop learning rate η1 to obtain the new model parameters; Repeat the above steps until the maximum number of iterations M is reached; Solidify the model parameters after the final convergence into the stage-independent initialization parameters θ * .

7. The meta-learning anomaly detection method for data throughout its life cycle according to claim 1, characterized in that: The generation phase adaptive anomaly detection model includes: For the target stage data stream collected in real time, a new supporting sample buffer S′ of this stage is constructed; Initialize the parameter θ * As the starting point, only the weights of the density estimation module are unfrozen, and n1 steps of mini-batch gradient update are performed on the new support sample buffer S′. The inner loop learning rate is set to η0, and η0 decreases exponentially with the batch; After each update, immediately evaluate the loss on the micro validation set V′ collected at the same time as the new support sample buffer S′. If the loss does not decrease for q consecutive times, stop updating; The model parameters obtained when the update stops are recorded as θ′, and the corresponding model is the stage-adaptive anomaly detection model.

8. The meta-learning anomaly detection method for data in the entire life cycle according to claim 1 or 7, characterized in that: The output of the final abnormal data set A includes: Calculate the anomaly score for each sample to be tested; Then the initial screening abnormality is determined based on the median deviation threshold Z1; Perform a stage consistency check on the samples with initial abnormalities, and confirm them as final abnormalities when the abnormality score exceeds the threshold Z2; For samples that are confirmed to be abnormal, their abnormal category, occurrence stage and timestamp are recorded and stored in the abnormal dataset A.

9. A meta-learning anomaly detection system for data throughout its life cycle, characterized in that: include: one or more processors; A memory storing operable instructions, wherein when the instructions are executed by the one or more processors, the one or more processors perform operations, wherein the operations include the process of the meta-learning anomaly detection method for data throughout its life cycle as described in any one of claims 1 to 8.

10. A computer-readable medium storing software, characterized in that: The software includes instructions that can be executed by one or more computers, and the instructions, through such execution, enable the one or more computers to perform operations, and the operations include the process of the meta-learning anomaly detection method under the entire data life cycle as described in any one of claims 1 to 8.

Images