A meta-learning anomaly detection method and system under a data full life cycle

Abstract

The application discloses a kind of data under the whole life cycle meta-learning anomaly detection method and system, comprising: according to the original data stream of pre-set sampling frequency acquisition, obtain original data set with stage label;Null-zero cleaning is carried out to it, and the output rough screening data set;According to life cycle stage, it is divided into several meta-learning tasks, and the model-independent meta-learning framework based on gradient is used to carry out outer loop and inner loop iteration with task as unit, obtain stage-independent initialization parameter, generate stage adaptive anomaly detection model;Calculate anomaly score to unlabeled query set and to-be-measured stream data, and recheck in combination with median deviation threshold, output final anomaly data set.The application is divided into meta-learning task by whole life cycle stage, and the model-independent meta-learning framework based on gradient is used to solve the problem that data characteristics are significantly different in different stages of data whole life cycle, anomaly detection model generalization ability is insufficient and difficult to adapt to interstage concept drift.

Description

Technical Field

[0001] This invention relates to the technical field of data processing and anomaly detection, and in particular to a meta-learning anomaly detection method and system for the entire data lifecycle. Background Technology

[0002] With the rapid development of big data, artificial intelligence and information system technologies, anomaly detection in each stage of data lifecycle has gradually become an important means to ensure data quality and stable system operation. In recent years, meta-learning, as a machine learning method that can achieve efficient learning with small samples and rapid adaptation to new tasks, has received widespread attention and research in the field of anomaly detection.

[0003] Existing meta-learning anomaly detection methods mostly use static or fixed model parameters. In actual production environments, when the system operating environment or data features undergo conceptual drift, the detection performance of the model will be significantly reduced, making it difficult to maintain high detection accuracy in the long term. At the same time, there is currently a lack of a unified stage-adaptive technology framework for anomaly detection in multi-stage data lifecycle scenarios. Typically, it is necessary to design a separate model for each stage, resulting in high development costs, slow response speed, and the inability to effectively share existing knowledge and parameters between stages.

[0004] CN114239712A discloses an anomaly detection method based on a meta-learning architecture for heterogeneous information networks. This method captures the structural information, heterogeneous features, and unlabeled information of heterogeneous information networks through graph neural networks, achieving effective transfer between different networks. CN119939445A discloses a few-shot anomaly detection method based on meta-learning and multimodal large models. It constructs a semantic-anomaly network through visual and language encoders to enhance the generalization ability and robustness of cross-category anomaly detection and improve anomaly detection accuracy. Although the above methods have achieved significant results in specific fields, they mainly focus on anomaly detection in static data scenarios or fixed-category tasks. They do not fully consider the differential characteristics and dynamic changes of data at different stages of the data lifecycle. Therefore, they are difficult to cope with the problem of rapid adaptation when there are significant differences in data features at different stages of the data lifecycle, which limits their application effect and generalization performance in complex information system scenarios.

[0005] Therefore, there is an urgent need to develop an anomaly detection technology that can take into account the differences in data characteristics at all stages of the entire lifecycle and can quickly and adaptively adjust when data characteristics drift, so as to meet the urgent need for real-time, robust, and high-precision anomaly detection in complex information systems.

[0006] In summary, existing anomaly detection technologies suffer from poor stage generalization ability, slow response to concept drift, and difficulty in effectively reusing knowledge from different stages of the data lifecycle. Our invention proposes a meta-learning anomaly detection method across the entire data lifecycle. By dividing meta-learning tasks into stages throughout the entire data lifecycle, employing a gradient-based model-independent meta-learning framework, and establishing a stage-adaptive online update mechanism, this method effectively addresses the issues of significant differences in data features across different stages of the data lifecycle, insufficient generalization ability of anomaly detection models, and difficulty in adapting to concept drift between stages. Summary of the Invention

[0007] The purpose of this section is to outline some aspects of the embodiments of the present invention and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section, as well as in the specification abstract and the title of the invention, to avoid obscuring the purpose of this section, the specification abstract, and the title of the invention. Such simplifications or omissions shall not be used to limit the scope of the invention.

[0008] In view of the aforementioned existing problems, the present invention is proposed.

[0009] To solve the above technical problems, the present invention provides the following technical solution: based on a preset sampling frequency, raw data streams are collected from the target information system in six stages of the data lifecycle: generation, transmission, storage, processing, archiving and destruction, to obtain a raw dataset D0 with stage labels;

[0010] The original dataset D0 is cleaned by null and zero values, and over-limit filtering is performed according to the adaptive upper and lower limit dynamic thresholds of each life cycle stage and the interquartile range criterion to output the coarse-screened dataset D1.

[0011] The coarse-screened dataset D1 is divided into several meta-learning tasks T according to the lifecycle stage. i For each task, extract the feature vector of each stage and label the supporting sample set S. i With unlabeled query set Q i ;

[0012] A gradient-based model-independent meta-learning framework is adopted, with the task T as an example. i Perform outer and inner loop iterations for each unit to obtain stage-independent initialization parameters θ. * ;

[0013] For the target stage data stream acquired in real time, the initialization parameter θ is used. * and the labeled support sample set S i Perform K-step gradient updates to generate a phase-adaptive anomaly detection model;

[0014] The anomaly detection model is used to analyze the unlabeled query set Q. iThe anomaly score is calculated based on the stream data to be tested, and then verified in conjunction with the median deviation threshold Z1 to output the final anomaly dataset A.

[0015] When the stage distribution drift metric is detected to exceed the preset threshold Z2, the incremental meta-learning process is triggered for retraining.

[0016] As a preferred embodiment of the meta-learning anomaly detection method under the data lifecycle described in this invention, obtaining the original dataset D0 with stage labels includes:

[0017] Data acquisition agent points are deployed in each of the six stages of the target information system: generation, transmission, storage, processing, archiving, and destruction.

[0018] Sampling is synchronously triggered at each data collection agent point based on a unified sampling time benchmark, and a unique timestamp is written for each sampling batch.

[0019] The sampled raw data is written to a temporary buffer, along with a stage label, data source node ID, sampling frequency, and sampling precision.

[0020] After passing the buffer consistency check, the appended original data is written to the original dataset D0 in batches, and the batch number is recorded.

[0021] The original dataset D0 includes at least the data timestamp, data value, data source node ID, corresponding lifecycle stage label, sampling frequency, and sampling accuracy information.

[0022] As a preferred embodiment of the meta-learning anomaly detection method throughout the data lifecycle described in this invention, the output coarse-screening dataset D1 includes:

[0023] Perform a field integrity check on the original dataset D0 and remove data rows that are missing key fields;

[0024] Fields that are detected as all zeros or empty strings are filled with missing markers and moved to the anomaly candidate table;

[0025] Based on the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion;

[0026] Data samples that do not exceed the limits and have complete fields are written into the coarse screening dataset D1, and samples that exceed the limits are moved into the abnormal candidate table.

[0027] As a preferred embodiment of the meta-learning anomaly detection method under the full data lifecycle described in this invention, the coarse-screened dataset D1 is divided into several meta-learning tasks T according to the lifecycle stages. i ,include:

[0028] Within each lifecycle stage, a sliding window of fixed length r is set and slides with a step size r1. The data in the window and its stage label form a candidate subset.

[0029] The candidate subset dataset is sorted according to a data volume threshold N. min Filtering and retaining a subset of data that meets the required data volume is a task T. i ;

[0030] For each task T i Write a unique task number, stage label, and window time range metadata.

[0031] As a preferred embodiment of the meta-learning anomaly detection method under the full data lifecycle described in this invention, feature vectors are extracted for each task stage, and the supporting sample set S is labeled. i With unlabeled query set Q i ,include:

[0032] The stage feature vector is composed of statistical features, frequency domain features and time series features. The statistical features include mean, variance, skewness and kurtosis. The frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum. The time series features include autocorrelation coefficient and volatility.

[0033] The labeled support sample set S i Stratified sampling was used to select normal samples and suspected abnormal samples, and the total number of samples |Si|≦m;

[0034] The unlabeled query set Q i Composed of the remaining data in the task window in chronological order, ensuring Q i With S i Non-overlapping and covers the entire window period.

[0035] As a preferred embodiment of the meta-learning anomaly detection method for the entire data lifecycle described in this invention, the acquisition of the stage-independent initialization parameter θ * ,include:

[0036] B tasks T are randomly selected from each lifecycle stage in sequence. i Form a batch of training tasks;

[0037] For each task T in the training task i In its labeled supporting sample set S i Perform n gradient updates to obtain temporary model parameters θ. i And calculate the temporary model parameter θ i In the corresponding unlabeled query set Q i The loss value L on i ;

[0038] Loss value L for all tasks i Calculate the average, and then perform a first-dimensional gradient update on the original model parameters θ according to the outer loop learning rate η1 to obtain the new model parameters.

[0039] Repeat the above steps until the maximum number of iterations M is reached;

[0040] The final converged model parameters are then fixed as stage-independent initialization parameters θ. * .

[0041] As a preferred embodiment of the meta-learning anomaly detection method throughout the entire data lifecycle described in this invention, the adaptive anomaly detection model in the generation stage includes:

[0042] For the target stage data stream acquired in real time, construct a new supporting sample buffer S′ for that stage;

[0043] Initialize parameter θ * Starting from the point of view, only the weights of the density estimation module are unfrozen, and n1 steps of mini-batch gradient update are performed on the new support sample buffer S′. The inner loop learning rate is set to η0, and η0 decreases exponentially with the batch size.

[0044] After each update, the loss is immediately evaluated on the mini validation set V′ collected concurrently with the new support sample buffer S′. If the loss does not decrease for q consecutive times, the update is stopped.

[0045] The model parameters obtained when the update stops are denoted as θ′, and the corresponding model is the stage-adaptive anomaly detection model.

[0046] As a preferred embodiment of the meta-learning anomaly detection method throughout the entire data lifecycle described in this invention, the output final anomaly dataset A includes:

[0047] Calculate an anomaly score for each sample to be tested;

[0048] Then, the initial screening is determined to be abnormal based on the median deviation threshold Z1;

[0049] For samples with abnormal initial screening, a phase consistency check is performed. When the abnormal score exceeds the threshold Z2, it is confirmed as a final abnormality.

[0050] For samples that are confirmed to be abnormal, record their abnormality category, occurrence stage, and timestamp, and store them in the abnormal dataset A.

[0051] As a preferred embodiment of the meta-learning anomaly detection system for the entire data lifecycle described in this invention, it includes: one or more processors;

[0052] The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the meta-learning anomaly detection method over the entire data lifecycle as described above.

[0053] As a preferred embodiment of the computer-readable medium for the storage software described in this invention, the software includes instructions executable by one or more computers, the instructions causing the one or more computers to perform operations, the operations including the flow of the meta-learning anomaly detection method throughout the data lifecycle as described above.

[0054] The beneficial effects of this invention are:

[0055] 1. By obtaining the original dataset with stage labels, accurate and systematic data collection of the target information system at each key stage is achieved, thus providing a comprehensive and detailed data foundation for anomaly detection. This effectively covers the entire data lifecycle and solves the problem that traditional anomaly detection methods only sample under specific stages or static data conditions. It improves the completeness and accuracy of the data acquisition process and enhances the reliability of anomaly detection from the source.

[0056] 2. By obtaining the coarse screening dataset, obvious anomalies and interference information in the original data are effectively removed, improving the accuracy and stability of subsequent analysis, avoiding false alarms caused by noisy data in the anomaly detection model, and achieving a high degree of data purification and stable data quality.

[0057] 3. By designing differentiated feature representations and task learning strategies based on the characteristics of data at different stages, the problem that traditional methods cannot fully express the heterogeneity of data at different stages is overcome, achieving the effect of clear tasks and accurate representation of data features;

[0058] 4. By obtaining stage-independent initialization parameters, the extraction and transfer of cross-stage anomaly detection knowledge are realized, which can be quickly generalized and adapted to new tasks at any stage of the life cycle. This solves the problem of poor generalization performance of traditional anomaly detection models, significantly improves the model's rapid adaptation ability and cross-stage generalization ability, reduces the training cost of new stage models, and improves deployment efficiency.

[0059] 5. By generating an adaptive anomaly detection model for the generation stage, the anomaly detection model can be rapidly fine-tuned online, thereby avoiding the problem of reduced accuracy due to stage changes or concept drift. It achieves the effect of real-time response to data changes and efficient and accurate adjustment, ensuring the online performance and reliability of the anomaly detection model.

[0060] 6. By further performing fine verification and neighborhood verification on the initially detected abnormal data, the problem of high false alarm rate in the existing technology is overcome, and the effect of highly reliable anomaly identification results and extremely low false alarm rate is achieved.

[0061] 7. By dynamically adjusting the model parameters, the problem of performance degradation over long-term use in existing anomaly detection technologies is effectively overcome. This achieves stable maintenance and proactive updating of anomaly detection accuracy under long-term use conditions, ensuring the long-term and stable application of anomaly detection technology in complex information system environments. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0063] Figure 1 This is a flowchart illustrating the meta-learning anomaly detection method throughout the entire data lifecycle as presented in this invention. Detailed Implementation

[0064] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0065] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without inventive effort should fall within the scope of protection of this invention.

[0066] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0067] According to an embodiment of the present invention, in combination Figure 1 The flowchart shown illustrates a meta-learning anomaly detection method across the entire data lifecycle, which specifically includes the following steps:

[0068] S1. Based on a preset sampling frequency, collect raw data streams from the target information system during the six stages of its data lifecycle: generation, transmission, storage, processing, archiving, and destruction, to obtain a raw dataset D0 with stage labels. Note that the following should be noted in this step:

[0069] Data acquisition agent points are deployed in each of the six stages of the target information system: generation, transmission, storage, processing, archiving, and destruction.

[0070] Sampling is synchronously triggered at each data collection agent point based on a unified sampling time benchmark, and a unique timestamp is written for each sampling batch.

[0071] The sampled raw data is written to a temporary buffer, along with a stage label, data source node ID, sampling frequency, and sampling precision.

[0072] After passing the buffer consistency check, the appended original data is written to the original dataset D0 in batches, and the batch number is recorded.

[0073] The original dataset D0 contains at least the data timestamp, data value, data source node ID, corresponding lifecycle stage label, sampling frequency, and sampling accuracy information.

[0074] As an example, each agent point uses a high-precision PTP (Precision Time Protocol) network clock (such as a master-slave clock system based on IEEE 1588-2008 Precision Time Protocol) to control the local system clock drift within ±1ms, and uses Coordinated Universal Time (UTC) as the unified time base to trigger sampling.

[0075] In an optional implementation, the preset sampling frequency is configured according to the service real-time level, with a default of 1Hz (once per second) and can be dynamically adjusted within the range of 0.1Hz to 10kHz.

[0076] The high real-time stage (generation and transmission) can be improved to ≥100Hz;

[0077] Low-frequency phases (archiving, destruction) can be reduced to ≤0.1Hz;

[0078] Frequency configuration is uniformly issued by the central scheduling module, and the sampling agent point automatically takes effect on the next full second after receiving the parameter update.

[0079] In an optional implementation, at the exact second UTC0ms, each agent point synchronously triggers a sampling, acquires the original data packet for the phase, and immediately writes it to a local temporary buffer. For example, the temporary buffer employs a write-delay parallel verification strategy, wherein:

[0080] Calculate whether the number of records in the batch matches the number of records configured for the node;

[0081] Verify that the timestamp difference for all entries is ≤1ms;

[0082] Verify whether the PhaseTag matches the node role by comparing the phase enumeration table.

[0083] Data batches that fail validation are automatically marked as Invalid and entered into the exception buffer queue, awaiting operation and maintenance audit.

[0084] Data batches that pass the consistency check are written to the original dataset D0 in batches within t+Δt (Δt≤200ms).

[0085] S2. Perform null-zero value cleaning on the original dataset D0, and perform over-limit filtering based on the adaptive upper and lower dynamic thresholds for each lifecycle stage and the interquartile range criterion, outputting the coarsely screened dataset D1. Note the following in this step:

[0086] Perform a field integrity check on the original dataset D0 and remove data rows that are missing key fields;

[0087] Fields that are detected as all zeros or empty strings are filled with missing markers and moved to the anomaly candidate table;

[0088] Based on the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion;

[0089] Write data samples that do not exceed the limits and have complete fields into the coarse screening dataset D1, and move the samples that exceed the limits into the abnormal candidate table.

[0090] As an example, extract the corresponding subset based on the lifecycle stage s∈{0,…,5} right Reference values ​​during the calculation phase (The median is used by default), based on the configuration management coefficient λ. s γ s (0<λ s γ s <1) Calculate the upper and lower dynamic thresholds:

[0091]

[0092] Among them, U s L is the upper limit threshold of stage s. s The lower threshold value for stage s;

[0093] For example, by means of Calculate the lower quartiles and the upper quartile get

[0094] Define the threshold coefficient κ as 1.5, and determine the threshold for each sample value v:

[0095]

[0096] Simultaneously determine whether v satisfies L. s ≤v≤U s ;

[0097] If any condition of v is not met, the corresponding record is moved into the exception candidate table E_cand and marked as out_of_range.

[0098] It should be noted that a record will only be written to the coarse-screened dataset D1 if it simultaneously meets the following conditions:

[0099] (1) Fields are complete;

[0100] (2) Non-empty value, non-zero value;

[0101] (3)L s ≤v≤U s And it is not an outlier in IQR;

[0102] For records written to the coarse screening dataset D1, retain the original six-element field {ts,value,node_id,phase,freq,precision}, and add qc_flag=0 to indicate that the quality inspection has passed;

[0103] Add qc_flag=1 and error_tag fields to the records transferred to the exception candidate table E_cand to identify the exception type for subsequent review.

[0104] S3. Divide the coarse-screened dataset D1 into several meta-learning tasks T according to the lifecycle stage. i For each task, extract the feature vector of each stage and label the supporting sample set S. i With unlabeled query set Q i In this step, it is important to note that:

[0105] Within each lifecycle stage, a sliding window of fixed length r is set and slides with a step size r1. The data in the window and its stage label form a candidate subset.

[0106] Candidate subsets are sorted by data volume threshold N min Filtering and retaining a subset of data that meets the required data volume is a task T. i ;

[0107] For each task T i Write a unique task number, stage label, and window time range metadata.

[0108] In an optional implementation, feature vectors for each task stage are extracted, and the supporting sample set S is labeled. i With unlabeled query set Q i ,include:

[0109] The stage feature vector is composed of statistical features, frequency domain features and time series features. The statistical features include mean, variance, skewness and kurtosis. The frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum. The time series features include autocorrelation coefficient and volatility.

[0110] Labeling support sample set S i Stratified sampling was used to select normal samples and suspected abnormal samples, and the total number of samples |Si|≦m;

[0111] Unlabeled query set Q i Composed of the remaining data in the task window in chronological order, ensuring Q i With S i Non-overlapping and covers the entire window period.

[0112] S4. Employ a gradient-based model-independent meta-learning framework for task T. i Perform outer and inner loop iterations for each unit to obtain stage-independent initialization parameters θ. * In this step, it is important to note that:

[0113] B tasks T are randomly selected from each lifecycle stage in sequence. i Form a batch of training tasks;

[0114] For each task T in the training task i In its labeled supporting sample set S i Perform n gradient updates to obtain temporary model parameters θ i And calculate the temporary model parameters θ i In the corresponding unlabeled query set Q i The loss value L on i ;

[0115] Loss value L for all tasks i Calculate the average, and then perform a first-dimensional gradient update on the original model parameters θ according to the outer loop learning rate η1 to obtain the new model parameters.

[0116] Repeat the above steps until the maximum number of iterations M is reached;

[0117] The final converged model parameters are then fixed as stage-independent initialization parameters θ. * .

[0118] S5. For the real-time acquired target stage data stream, use the initialization parameter θ * and labeled supporting sample set S i Perform K-step gradient updates to generate a stage-adaptive anomaly detection model. Note the following in this step:

[0119] When the real-time data stream of the target stage reaches w consecutive sampling periods, the system creates a new support sample buffer S′ in memory, in which the ratio of normal samples to suspected abnormal samples is not less than 3:1, and the upper limit of the buffer size |S′| is dynamically issued by the central scheduling module.

[0120] Simultaneously, v samples are randomly selected from the same time window to form a validation set V′, which is locked until the end of this round of online updates;

[0121] Initialize parameter θ with offline phase independence * Starting with the density estimation submodule G, only the weights of the density estimation submodule G are unfrozen, while the weights of the remaining feature embedding submodule E are kept frozen to reduce the computational cost of online fine-tuning.

[0122] S′ is divided into segments according to batch size b. A small batch size, with an inner loop learning rate η0;

[0123] Perform forward-backward propagation once for each mini-batch and update the parameters of G, for a total of n1 steps;

[0124] After each update step is completed, the loss value is immediately calculated on the validation set V′. If the loss value does not decrease significantly within q consecutive iterations, the update is stopped.

[0125] The parameters at which updates stop are denoted as θ′ and written into the model registry, generating a version number ver=<phase,time> ;

[0126] Its corresponding model f θ′ This is the stage-adaptive anomaly detection model, which can be directly called in step S6.

[0127] As an example, the mathematical expression for the stage-adaptive anomaly detection model is:

[0128]

[0129] Where x is the sample vector to be tested, θ′ is the set of model parameters after online update, σ is the Logistic compression function, Z is the normalization constant, T is the buffer window length, and Ψ α (x, τ) is a nonlinear mapping with time offset, Φ β (·) is the logarithmic gain function for logarithmic modulation, s i To support the feature vector of sample i, κ i η is the kernel density weight, η is the adaptive scale of the Gaussian kernel bandwidth, γ is the stabilization constant to prevent the denominator from being zero, d is the feature dimension, and δ is used to suppress the growth of extreme integral values.

[0130] It should be noted that when f θ′ When (x) ≤ 0.5, it is a normal sample; when 0.5 <f θ′If (x) ≤ 0.8, it is a suspicious sample and needs to be further verified with threshold Z1 in step S6; when f θ′ If (x) > 0.8, it is directly marked as a high-confidence anomaly.

[0131] S6. Use an anomaly detection model to analyze the unlabeled query set Q. i Anomaly scores are calculated from the stream data to be tested, and then verified using the median deviation threshold Z1, resulting in the final anomaly dataset A. Note that the following points should be noted in this step:

[0132] Call the stage adaptive anomaly detection model f obtained in step S5 θ′ For the unlabeled query set Q i And real-time test stream data is input one by one, and the original score p(x)∈(0,1) is output;

[0133] Construct a comprehensive anomaly score by combining support and confidence interval:

[0134]

[0135] in, ξ is the mean of p(x) within the current time window, ξ>0 is the Sigmoid steepness scheduling coefficient, and Score(x) is the comprehensive anomaly score;

[0136] Calculate the comprehensive score set {Score(x)} within the same sliding window W j The median Med and absolute median difference MAD of the score are calculated as follows: (|Score(x)) j )-Med|);

[0137] Let the median deviation threshold Z1 = Med + β·MAD, β∈[2,4];

[0138] When Score(x) > Z1, sample x is marked as an anomaly in the initial screening;

[0139] For the initial screening of abnormal samples, a ring neighbor sequence is established according to their phase label. If Score(x) > Z2 (global high confidence threshold, Z2∈(0.85,0.95)), then sample x is confirmed as the final abnormal.

[0140] Otherwise, it will be downgraded to suspicious and re-examined in the next window;

[0141] For confirmed abnormal samples, record the "timestamp ts, source node node_id, lifecycle stage phase, and overall score". Generate a unique anomaly ID by specifying the anomaly level (high / medium) and the initial screening reason (MAD or high confidence Z2) and write it into the anomaly dataset A.

[0142] The abnormal dataset A is stored in columnar format and partitioned by phase→ts. An incremental backup is triggered every m records written, and the newly added abnormal ID and phase are pushed to the monitoring system through the alarm bus to achieve real-time early warning.

[0143] S7. When the stage distribution drift metric δ is detected to exceed the preset threshold Z2, the incremental meta-learning process is triggered for retraining. Note that the following points should be noted in this step:

[0144] Set the length to w for each lifecycle phase s. R Reference window R s and open a length of w M Motion detection window M s When the detection window is filled with the latest sample, the stage distribution drift metric is calculated using the maximum mean difference Δs.

[0145] Configure early warning thresholds for each stage With forced threshold

[0146] like This indicates a stable state, where only the detection window is scrolled.

[0147] like This indicates a slight drift state, in which case a rapid fine-tuning process will be executed.

[0148] like This indicates a severe drift state, and an incremental retraining process will be executed.

[0149] As an example, performing a quick fine-tuning process during a slight drift includes the following steps:

[0150] From the current stage, the most recent w M Construct an incremental support set S from each sample. r With the validation set V r ;

[0151] Based on the current stage adaptive model f θ′ Starting with the density estimation module, only the weights of the last h layers are unfrozen, using a learning rate η. fast Perform small-batch updates;

[0152] If V r The loss decreased by ≥ ∈ compared to before the update fast That is, accept the new parameters, otherwise roll back and retain the old model;

[0153] Update Reference Window R s For the most recent w R One stable sample; record model version ver fast And monitoring was restored.

[0154] As an example, the incremental retraining process under severe drift conditions includes the following steps:

[0155] Within stage s, window w R Re-splitting the coarse-screened data into units and generating a temporary task set.

[0156] since Randomly select B inc Task and Historical Task Library T hist B in hist The steady-state tasks are concatenated into an incremental training batch. inc ;

[0157] New initialization parameters Test on the independent validation task set in stage s. If the average AUC ≥ AUC min And it improves upon the old initialization by ≥δ gain Then replace the global initialization parameters. Otherwise, keep the original value and record the alarm;

[0158] After successful replacement, a version number ver is generated. inc And distribute to online nodes;

[0159] If q follows fail If a detection window continuously triggers severe drift, it will automatically roll back to the previous stable version. prev .

[0160] It should be noted that drift detection and fine-tuning / retraining jobs are triggered by the central orchestrator through an asynchronous event queue. All computational resource consumption, time consumption, and version evolution information are written to the log table. When the memory or GPU usage of a retraining job exceeds a preset threshold, it is dynamically downgraded to a sequential task to ensure the priority of online inference.

[0161] Preferably, the present invention achieves continuous quantitative monitoring and hierarchical response to changes in data distribution during the lifecycle stages. Mild drifts are quickly fine-tuned to maintain low latency, while high-intensity drifts trigger incremental meta-learning to ensure long-term model robustness. At the same time, it relies on dual thresholds and rollback mechanisms to avoid frequent oscillations and ensure the stability of the anomaly detection service.

[0162] The aforementioned methods for extracting feature vectors, labeled support sample sets, and unlabeled query sets, as well as the loss value L... i The calculation method can be carried out using existing technologies and methods, which will not be elaborated in this example.

[0163] In addition to the above embodiments, other aspects of the present invention also propose a meta-learning anomaly detection system for the entire data lifecycle, comprising: one or more processors and a memory.

[0164] The memory is used to store operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of a meta-learning anomaly detection method throughout the data lifecycle of the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0165] Other aspects disclosed in the embodiments of the present invention also propose a computer-readable medium for storing software including instructions executable by one or more computers, which, upon execution, cause the one or more computers to perform operations including the flow of a meta-learning anomaly detection method throughout the data lifecycle of the foregoing embodiments, particularly... Figure 1 The flowchart of the method is shown.

[0166] It should be recognized that embodiments of the present invention may be implemented or carried out by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable storage medium.

[0167] The method can be implemented using standard programming techniques, including a non-transitory computer-readable storage medium configured with a computer program in the computer program, wherein the storage medium is configured such that the computer operates in a specific and predefined manner.

[0168] Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system; however, if required, the program can be implemented in assembly or machine language.

[0169] In any case, the language can be either compiled or interpreted.

[0170] Furthermore, for this purpose, the program can run on programmed application-specific integrated circuits.

[0171] The processes described herein (or variations and / or combinations thereof) can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that commonly executes on one or more processors. The computer program includes a plurality of instructions executable by one or more processors.

[0172] Furthermore, the method can be implemented in any suitable computing platform, including but not limited to personal computers, minicomputers, mainframes, workstations, networked or distributed computing environments, standalone or integrated computer platforms, or in communication with charged particle tools or other imaging devices.

[0173] Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether portable or integrated into a computing platform, such as a hard disk, optical read and / or write storage medium, RAM, ROM, etc., such that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the processes described herein.

[0174] Furthermore, machine-readable code, or parts thereof, can be transmitted via wired or wireless networks.

[0175] When such media includes instructions or programs that combine with a microprocessor or other data processor to implement the steps described above, the invention described herein includes these and other different types of non-transitory computer-readable storage media.

[0176] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A meta-learning anomaly detection method throughout the entire data lifecycle, characterized in that, include: Based on the preset sampling frequency, raw data streams are collected from the target information system in six stages of the data lifecycle: generation, transmission, storage, processing, archiving, and destruction, to obtain the raw dataset D0 with stage labels. The original dataset D0 is cleaned by null and zero values, and over-limit filtering is performed according to the adaptive upper and lower limit dynamic thresholds of each life cycle stage and the interquartile range criterion to output the coarse-screened dataset D1. The coarse-screened dataset D1 is divided into several meta-learning tasks T according to the lifecycle stage. i For each task, extract the feature vector of each stage and label the supporting sample set S. i With unlabeled query set Q i ; A gradient-based model-independent meta-learning framework is adopted, with the task T as an example. i Perform outer and inner loop iterations for each unit to obtain stage-independent initialization parameters. ; For the target stage data stream acquired in real time, the initialization parameters are used. and the labeled support sample set S i Perform K-step gradient updates to generate a phase-adaptive anomaly detection model; The adaptive anomaly detection model for the generation stage includes: constructing a new support sample buffer S′ for the target stage data stream acquired in real time; and initializing parameters. Starting with the density estimation module weights, only the new support sample buffer S′ is unfrozen, and n1 mini-batch gradient updates are performed on the new support sample buffer S′, with the inner loop learning rate set to 1. ,and The batch exponent decreases; after each update step, the loss is immediately evaluated on the mini validation set V′ collected at the same time as the new support sample buffer S′. If the loss does not decrease for q consecutive times, the update is stopped; the model parameters obtained when the update is stopped are denoted as θ′, and the corresponding model is the stage adaptive anomaly detection model. The anomaly detection model is used to analyze the unlabeled query set Q. i The anomaly score is calculated based on the stream data to be tested, and then verified in conjunction with the median deviation threshold Z1 to output the final anomaly dataset A. When the stage distribution drift metric is detected to exceed the preset threshold Z2, the incremental meta-learning process is triggered for retraining.

2. The meta-learning anomaly detection method throughout the entire data lifecycle as described in claim 1, characterized in that, The process of obtaining the original dataset D0 with stage labels includes: Data acquisition agent points are deployed in each of the six stages of the target information system: generation, transmission, storage, processing, archiving, and destruction. Sampling is synchronously triggered at each data collection agent point based on a unified sampling time benchmark, and a unique timestamp is written for each sampling batch. The sampled raw data is written to a temporary buffer, along with a stage label, data source node ID, sampling frequency, and sampling precision. After passing the buffer consistency check, the appended original data is written to the original dataset D0 in batches, and the batch number is recorded. The original dataset D0 includes at least the data timestamp, data value, data source node ID, corresponding lifecycle stage label, sampling frequency, and sampling accuracy information.

3. The meta-learning anomaly detection method throughout the entire data lifecycle according to claim 1 or 2, characterized in that, The output coarse screening dataset D1 includes: Perform a field integrity check on the original dataset D0 and remove data rows that are missing key fields; Fields that are detected as all zeros or empty strings are filled with missing markers and moved to the anomaly candidate table; Based on the reference value V of the current life cycle stage ref Calculate adaptive upper and lower thresholds and detect out-of-limit data based on the interquartile range criterion; Data samples that do not exceed the limits and have complete fields are written into the coarse screening dataset D1, and samples that exceed the limits are moved into the abnormal candidate table.

4. The meta-learning anomaly detection method throughout the entire data lifecycle as described in claim 3, characterized in that, The coarse-screened dataset D1 is divided into several meta-learning tasks T according to the lifecycle stage. i ,include: Within each lifecycle stage, a sliding window of fixed length r is set and slides with a step size r1. The data in the window and its stage label form a candidate subset. The candidate subset dataset is sorted according to a data volume threshold N. min Filtering and retaining a subset of data that meets the required data volume is a task T. i ; For each task T i Write a unique task number, stage label, and window time range metadata.

5. The meta-learning anomaly detection method throughout the entire data lifecycle as described in claim 4, characterized in that, For each task, extract the feature vector of each stage and label the supporting sample set S. i With unlabeled query set Q i ,include: The stage feature vector is composed of statistical features, frequency domain features and time series features. The statistical features include mean, variance, skewness and kurtosis. The frequency domain features are the amplitudes of the first three main peaks of the Fourier spectrum. The time series features include autocorrelation coefficient and volatility. The labeled support sample set S i Stratified sampling was used to select normal samples and suspected abnormal samples, and the total number of samples |Si|≦m; The unlabeled query set Q i Composed of the remaining data in the task window in chronological order, ensuring Q i With S i Non-overlapping and covers the entire window period.

6. The meta-learning anomaly detection method throughout the entire data lifecycle according to claim 1, characterized in that, The initialization parameters are independent of the acquisition stage. ,include: B tasks T are randomly selected from each lifecycle stage in sequence. i Form a batch of training tasks; For each task T in the training task i In its labeled supporting sample set S i Perform n gradient updates to obtain temporary model parameters. And calculate the temporary model parameters. In the corresponding unlabeled query set Q i loss value on ; Loss value for all tasks Calculate the average based on the outer loop learning rate. For the original model parameters Perform a primary gradient update to obtain the new model parameters; Repeat the above steps until the maximum number of iterations M is reached; The final converged model parameters are then fixed as stage-independent initialization parameters. .

7. The meta-learning anomaly detection method throughout the entire data lifecycle according to claim 1, characterized in that, The final output anomaly dataset A includes: Calculate an anomaly score for each sample to be tested; Then, the initial screening is determined to be abnormal based on the median deviation threshold Z1; For samples with abnormal initial screening, a phase consistency check is performed. When the abnormal score exceeds the threshold Z2, it is confirmed as a final abnormality. For samples that are confirmed to be abnormal, record their abnormality category, occurrence stage, and timestamp, and store them in the abnormal dataset A.

8. A meta-learning anomaly detection system throughout the entire data lifecycle, characterized in that, include: One or more processors; The memory stores operable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, including the flow of the meta-learning anomaly detection method over the entire data lifecycle as described in any one of claims 1 to 7.

9. A computer-readable medium for storing software, characterized in that: The software includes instructions executable by one or more computers, which cause the one or more computers to perform operations, including the flow of the meta-learning anomaly detection method for the entire data lifecycle as described in any one of claims 1 to 7.

Images