Data anomaly detection method, system and equipment based on size model and medium

Abstract

The invention discloses a large and small model-based data anomaly detection method, system and device and a medium, belongs to the technical field of data processing and artificial intelligence, and aims to solve the technical problem of how to combine the rapid screening capability of a small model and the deep analysis capability of a large model to realize efficient and collaborative anomaly detection and cleaning. According to the technical scheme, an improved Isolation Forest small model is used for rapid screening, potential anomalies in large-scale data are rapidly detected through a dynamic threshold adjustment mechanism, adaptive feature weight distribution and multi-scale division optimization, an anomaly score is generated, and suspected abnormal data are preliminarily screened; secondly, performing fine detection on suspected abnormal data by utilizing a large model, and accurately identifying an abnormal type and generating a high-reliability abnormal result through deep semantic embedding analysis, abnormal feature clustering and domain rule verification; and finally detected abnormal data are classified and labeled, and the data characteristics are combined to process the abnormal data by adopting a deletion, correction or marking cleaning strategy.

Description

Technical Field

[0001] The present invention relates to the field of data processing and artificial intelligence technology, and in particular to a method, system, device and medium for detecting data anomalies based on a size model. Background Art

[0002] With the rapid development of artificial intelligence technology, the processing and application of large-scale data has become a key task in many fields. However, data quality issues (such as outliers, noisy data, incomplete data, etc.) directly affect the training effect of subsequent models and the accuracy of decision-making. Anomaly detection and data cleaning, as the core link of data preprocessing, have become an important direction in current research and application.

[0003] Existing anomaly detection technologies can be roughly divided into rule-based methods and model-based methods. Rule-based methods rely on rule sets or specific statistical characteristics formulated by domain experts, such as setting fixed thresholds or using simple statistical indicators for detection. Although such methods are intuitive and simple, they lack sufficient adaptability to data anomalies in complex scenarios and have limited performance when processing multidimensional data or dynamic distributions. Model-based methods use machine learning and deep learning technologies to automatically capture abnormal patterns of data by training models. Common ones include Isolation Forest, One-Class SVM, and autoencoders based on deep learning. These methods have made certain breakthroughs in the efficiency and flexibility of anomaly detection, but there are still obvious problems in large-scale data scenarios: lack of coordination between rapid screening and deep analysis, resulting in limited performance; lack of adaptability to data diversity and domain characteristics, and relatively simple means of handling abnormal results, which makes it difficult to meet the needs of complex scenarios.

[0004] In recent years, with the widespread application of pre-trained large models, their powerful capabilities in semantic understanding and feature extraction have provided a new direction for data anomaly detection. However, large models face the problems of high computational cost and low efficiency when directly processing large-scale data, while small models have advantages in speed but are insufficient in detection accuracy and domain adaptability.

[0005] Therefore, how to combine the rapid screening capabilities of small models and the deep analysis capabilities of large models to achieve efficient and coordinated anomaly detection and cleaning is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical task of the present invention is to provide a data anomaly detection method, system, device and medium based on large and small models to solve the problem of how to combine the rapid screening capability of small models and the deep analysis capability of large models to achieve efficient and coordinated anomaly detection and cleaning.

[0007] The technical task of the present invention is achieved in the following manner. A data anomaly detection method based on a size model first uses an improved Isolation Forest small model for rapid screening. Through a dynamic threshold adjustment mechanism, adaptive feature weight allocation, and multi-scale division optimization, it quickly detects potential anomalies in large-scale data and generates anomaly scores to preliminarily screen suspected anomaly data. Then, it uses a large model to finely detect the suspected anomaly data. Through in-depth semantic embedding analysis, anomaly feature clustering, and domain rule verification, it accurately identifies the anomaly types and generates highly credible anomaly results. It also classifies and labels the finally detected anomaly data, and adopts cleaning strategies such as deletion, correction, or marking in combination with data characteristics for processing. At the same time, through a feedback optimization mechanism, it further dynamically adjusts the parameters of the small model and the detection strategy of the large model to form an efficient and collaborative anomaly detection and cleaning closed-loop. Specifically as follows:

[0008] Data preprocessing and feature generation: Clean, normalize, and extract features from the input data to obtain preprocessed data, ensuring the quality of the input data and the accuracy of feature expression. Among them, feature extraction includes extracting explicit features through domain knowledge rules and generating implicit features through an autoencoder.

[0009] Rapid screening: Through the improved Isolation Forest algorithm, combined with dynamic threshold adjustment, multi-scale division, and feature weight allocation techniques, preliminarily screen the preprocessed data for anomalies and quickly mark suspected anomaly data.

[0010] Deep detection: Finely detect the screened suspected anomaly data, and use a large model of a pre-trained language model or a deep embedding model for semantic embedding analysis, anomaly feature clustering, and rule verification to comprehensively identify anomaly data under complex patterns and classify the anomalies.

[0011] Anomaly data cleaning: Combining the efficient marking ability of the small model and the semantic analysis ability of the large model, adopt cleaning strategies of deletion, correction, or annotation for noise data, potential error data, and out-of-domain data respectively, and dynamically adjust to adapt to multi-scenario requirements to ensure data quality.

[0012] Feedback optimization and closed-loop regulation: Dynamically adjust the parameters and strategies of each stage, and continuously improve the performance and enhance the adaptability in diverse scenarios through real-time monitoring of the detection and cleaning effects. Among them, dynamically adjusting the parameters and strategies of each stage includes optimizing the threshold of rapid screening, the rule base of the large model, and the cleaning strategy.

[0013] Preferably, the data preprocessing and feature generation are specifically as follows:

[0014] The input data is cleaned through standardized processes; the cleaning operations include removing redundant information, correcting abnormal formats, and filling missing values; for noisy data, a strategy combining rule-driven filtering methods and statistical analysis is used to detect the skewness, kurtosis and distribution characteristics of the data, identify data points that may have deviations and make preliminary corrections; at the same time, according to the structural characteristics of the data (such as text, images, tables, etc.), corresponding preprocessing methods are used, specifically: for text data, word segmentation, stop word removal and stem extraction techniques are used; for numerical data, range normalization or logarithmic transformation is performed to reduce the impact of numerical differences on the model;

[0015] After the input data is cleaned, a combination of explicit feature extraction based on domain rules and data-driven implicit feature learning is used to provide multi-dimensional feature expression. Specifically, on the one hand, key attributes are automatically identified and explicit features are generated through domain knowledge, such as extracting core indicators such as transaction frequency and amount fluctuation in financial data; on the other hand, implicit features are generated using autoencoders or deep embedding models to capture the deep semantics and potential patterns of the data. The combination of explicit and implicit features ensures the comprehensiveness and effectiveness of data representation.

[0016] The extracted features are screened through the feature importance evaluation mechanism; specifically: the feature set is dynamically adjusted by combining statistical methods (such as chi-square test and variance analysis) and model-driven methods (such as gradient-based feature importance scoring) to reduce the interference of redundant features on subsequent models; at the same time, in order to support the efficient operation of rapid screening, the weight wi of each feature is further calculated, and the weight of each feature is used to guide the splitting and judgment operations of the small model in preliminary detection to ensure the optimization of resource allocation; after data preprocessing and feature generation, the original data is converted into a high-quality, structured data set with task-related features, which provides a reliable input basis for anomaly detection and improves the efficiency and accuracy of subsequent modules.

[0017] As a preferred method, fast screening is based on the Isolation Forest algorithm with multiple improvements on dynamic threshold adjustment, multi-scale partitioning mechanism, and task-specific feature weight allocation, which improves the measurement efficiency and sensitivity to complex abnormal patterns; and the improved Isolation Forest algorithm introduces a feature weight-driven multi-scale partitioning mechanism when constructing the split tree, as follows:

[0018] When traditional algorithms randomly select a single feature for splitting, they may ignore the influence of key features, resulting in insufficient separation of abnormal data. By calculating the importance weight wi of the feature and combining it with a dynamic sampling strategy, features with higher weights are given priority during splitting to ensure the contribution of key features to anomaly detection. The weight distribution calculation formula is as follows:

[0019]

[0020] Among them, Imp(f i ) represents the importance score of feature f i , which is determined based on statistical characteristics (such as variance, information gain) or domain rules; n represents the total number of features;

[0021] Adopt a dynamic threshold adjustment mechanism to dynamically set the anomaly score threshold θ according to the distribution characteristics of the input data (such as mean, standard deviation, and skewness); during the detection process, the Isolation Forest algorithm calculates the path length L(x) for each data point x and generates an anomaly score S(x), and the formula is as follows:

[0022]

[0023] Among them, is the expected value of the path length of normal data;

[0024] Calculate the dynamic threshold θ = μ(S) + α·σ(S) according to the data distribution;

[0025] Among them, μ(S) and σ(S) represent the mean and standard deviation of the anomaly score S respectively; α represents an adjustment parameter for controlling the detection sensitivity; when S(x) ≥ θ, the data point is marked as a suspected anomaly;

[0026] To improve the interpretability of abnormal data, after the screening is completed, preliminary annotation is carried out on the suspected abnormal points, and the distribution deviation of the suspected abnormal points in different feature dimensions is analyzed: by recording the feature dimensions and their path depths at which the abnormal points are separated in the splitting tree, the characteristic description of the abnormal points is generated, providing input information for subsequent in-depth detection.

[0027] Preferably, the in-depth detection is as follows:

[0028] Use pre-trained large models such as BERT or GPT to perform semantic embedding analysis on each piece of suspected abnormal data, convert the data into high-dimensional semantic vectors, and the semantic vectors can capture the context correlation and deep features of the data; by comparing the similarity with the normal data distribution, evaluate the deviation degree of the data point and preliminarily judge the possibility of anomaly; at the same time, by analyzing the most significant deviation dimensions in the semantic vectors, identify potential patterns of anomalies, such as abnormal behaviors or distribution anomalies in specific domains;

[0029] After completing the embedding analysis, perform feature clustering on the suspected abnormal data: Based on the distribution density of the embedding vectors, use a density clustering algorithm (such as DBSCAN) to group the abnormal data to identify the abnormal patterns of the data; for example, in financial transaction data, specific patterns such as "high-frequency trading anomaly" and "sudden increase in amount anomaly" may be identified. The clustering results not only provide a basis for the cleaning module but also enhance the interpretability of the detection results;

[0030] Validate the abnormal data in combination with domain rules to further improve the reliability of the detection; the rule base consists of industry-specific abnormal patterns defined by experts. For example, identify abnormal diagnostic results in medical data and detect logical errors in legal documents. By comparing the detection results with the rule base, it is possible to verify whether the abnormal data conforms to known patterns and provide a clear judgment criterion for the model;

[0031] To ensure the comprehensiveness and accuracy of the detection, an abnormal credibility scoring mechanism is also adopted to comprehensively evaluate the abnormal nature of each piece of data. The score is generated based on the semantic deviation degree, clustering results, and rule matching scores. Priority is given to processing abnormal data with high confidence, and data with low confidence is reserved for subsequent manual review or supplementary detection;

[0032] By combining semantic embedding, clustering analysis, and rule verification, give full play to the advantages of the large model in identifying complex abnormal patterns, provide multi-level judgment and classification for suspected abnormal data, provide high-quality input data for the subsequent cleaning module, and at the same time enhance the detection ability in complex data scenarios.

[0033] Preferably, the abnormal data cleaning is as follows:

[0034] Classify and label the abnormal data marked by rapid screening and in-depth detection. The abnormal data is divided into three categories: noise data, potential error data, and out-of-domain data; among them, noise data includes obviously invalid records, and obviously invalid records include missing values, incomplete fields, or meaningless characters; potential error data refers to data points that may deviate from the normal range due to input errors or system anomalies, including large values with abnormal transaction amounts or mutation points in time series; out-of-domain data is data that does not conform to the current task background but may have potential analysis value, such as irrelevant information input across industries;

[0035] Develop differentiated cleaning strategies for different categories of abnormal data. Among them, for noise data, the small model directly deletes or fills it through quick scoring. For example, use rule-driven correction methods to interpolate or fill missing values, and eliminate completely invalid data points to quickly complete the processing. The processing of potential error data depends on the in-depth analysis ability of the large model. Use the semantic understanding of the large model to perform context analysis on the data, and combine the rule base to verify the legality of the data. Specifically: for abnormal transaction amounts, the large model generates reasonable correction values through context fitting and verifies whether the corrected data conforms to domain rules; for mutation points in time series, use trend prediction and interpolation repair methods to ensure continuity and logic in the time dimension; for out-of-domain data, the large model further analyzes the potential characteristics of out-of-domain data, compares it with the normal data distribution through semantic embedding, extracts key deviation information and generates annotations to mark the data for subsequent manual or automated analysis, avoiding misdeleting information that may have potential value.

[0036] Design a dynamic cleaning mechanism to adjust the strategy in real time according to the quality of the cleaned data and user feedback. For example, if a certain type of noise data appears frequently, optimize the rule-driven cleaning process to improve processing efficiency; if valuable patterns are found in out-of-domain data, record the feature deviations and give priority to retaining similar data in subsequent cleaning. The dynamic adjustment ability enables the system to adapt to diverse scenario requirements.

[0037] After cleaning, verify the consistency and integrity of the cleaning results to ensure that the quality of the corrected dataset meets the standard. The data after anomaly processing is reintegrated with the normal data to form a high-quality output dataset; and generate a cleaning report, which details the anomaly classification, correction or marking operations, providing users with a transparent cleaning process. Among them, the cleaning report includes classification statistics of anomaly detection, records of cleaning operations and performance evaluation indicators; the detailed information recorded in the report covers the quality and context relevance of the cleaned data for users to refer to or verify.

[0038] More preferably, the feedback optimization and closed-loop regulation are as follows:

[0039] Collect feedback data at each stage, covering the screening accuracy of the quick screening module, the classification accuracy of in-depth detection, and the effect evaluation of cleaning. The specific indicators include the missed detection rate and false detection rate of quick screening, the accuracy of anomaly type classification by in-depth detection, the integrity, consistency and context relevance of the data after cleaning.

[0040] A data anomaly detection system based on large and small models, the system includes:

[0041] Data preprocessing and feature generation module, which is used to clean, normalize and extract features from the input data to generate explicit features and implicit features; among them, the explicit features are extracted based on domain rules, and the implicit features are generated through autoencoders or deep embedding models;

[0042] Fast screening module, which is used to preliminarily screen the data based on the improved Isolation Forest algorithm, and improve the ability to quickly identify abnormal data through dynamic threshold adjustment, multi-scale partitioning and feature weight allocation mechanisms;

[0043] Deep detection module, which is used to conduct refined analysis on the suspected abnormal data screened out by combining with a large model, generate high-dimensional feature representations through semantic embedding, and achieve abnormal classification through clustering analysis and rule verification;

[0044] Abnormal data cleaning module, which is used to adopt a classification cleaning strategy for noise data, potential error data and out-of-domain data based on the efficient processing of a small model and the deep semantic analysis of a large model; among them, the classification cleaning strategy includes deletion, correction and annotation;

[0045] Feedback optimization and closed-loop adjustment module, which is used to dynamically adjust the parameters and strategies in each stage to form a closed-loop optimization process from anomaly detection to cleaning.

[0046] Preferably, the data preprocessing and feature generation module combines a rule-driven cleaning method with statistical analysis to correct or eliminate noise data; among them, the feature generation process combines statistical methods and model-driven methods to dynamically adjust the feature set and optimize the weight allocation of explicit features and implicit features to support the efficient operation of the fast screening module;

[0047] The fast screening module is based on the improved Isolation Forest algorithm, optimizes the data splitting strategy with the support of multi-scale partitioning by dynamically adjusting the threshold and combining the feature weight allocation mechanism; at the same time, dynamically sets the anomaly score threshold according to the data distribution to ensure the accuracy and resource allocation efficiency of fast screening, and generates a description of the characteristics of anomaly points for subsequent deep detection;

[0048] The deep detection module generates high-dimensional semantic embeddings through a large model to capture the deep semantic features of abnormal data, and classifies the abnormal data by combining the density clustering algorithm and domain rule verification; and prioritizes the detection results through the anomaly credibility score, so as to improve the accuracy and interpretability of anomaly classification;

[0049] The abnormal data cleaning module adopts different cleaning strategies for different types of abnormal data: directly correct or eliminate noise data, generate correction values through large model analysis for potential error data and verify their legality, and perform semantic deviation analysis on out-of-domain data and generate annotation marks, so as to ensure the quality and integrity of the data after cleaning;

[0050] The feedback optimization and closed-loop regulation module generates feedback signals by collecting the performance indicators of the quick screening module, the in-depth detection module and the cleaning module, and dynamically adjusts the parameters and strategies of each module; and expands or optimizes the rule base for misjudged abnormal types, supports lightweight fine-tuning of the large model, and finally forms a closed-loop optimization process adapted to different data distributions and domain scenarios.

[0051] An electronic device, comprising: a memory and at least one processor;

[0052] Wherein, a computer program is stored on the memory;

[0053] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the data anomaly detection method based on the large and small models as described above.

[0054] A computer-readable storage medium, in which a computer program is stored, and the computer program can be executed by a processor to implement the data anomaly detection method based on the large and small models as described above.

[0055] The data anomaly detection method, system, device and medium based on the large and small models of the present invention have the following advantages:

[0056] (1) Through modular design, the present invention supports preprocessing, detection and cleaning operations for different data types (including text, images and numerical tables), taking into account both speed and accuracy, meeting the needs of diverse data scenarios, and can be applied to fields such as financial risk control, medical data processing, industrial monitoring and e-commerce recommendation, etc. It has wide cross-domain applicability, significantly improves the efficiency and quality of data processing, provides strong support for intelligent decision-making and data management, and can provide flexible, accurate and efficient solutions for abnormal patterns in different fields;

[0057] (2) The present invention realizes quick screening through the improved Isolation Forest, can efficiently mark suspected abnormal data in large-scale data processing, and at the same time uses a large model for in-depth detection to accurately identify abnormalities under complex patterns; the small model is responsible for efficient preliminary screening, and the large model focuses on fine analysis. The two work together to effectively reduce the computational cost and significantly improve the accuracy of anomaly detection;

[0058] (3) The present invention adopts a flexible anomaly cleaning strategy: The anomaly data cleaning module combines the efficient marking of small models with the deep semantic understanding ability of large models, and adopts different cleaning strategies for different types of anomalies (noise data, potential error data, out-of-domain data), including deletion, correction, and annotation; Dynamically adjusting the cleaning can adapt to diverse scenarios, ensuring both data quality and retaining potentially useful information;

[0059] (4) The present invention has dynamic optimization and self-adaptive capabilities: Through the feedback optimization and closed-loop regulation module, the parameters at each stage (such as screening thresholds, feature weights, rule bases) are dynamically adjusted to form a closed-loop optimization mechanism from detection to cleaning; As the system runs, it can continuously improve the adaptability to new data distributions and anomaly patterns, and improve the detection and cleaning effects;

[0060] (5) The present invention can perform comprehensive data preprocessing and feature generation: The data preprocessing module combines domain rules and autoencoders to generate explicit and implicit features, ensuring the comprehensiveness and accuracy of feature representation. The importance evaluation and screening mechanism for multi-dimensional features reduces redundant information, provides high-quality input for the fast screening module, and improves the overall efficiency of subsequent modules;

[0061] (6) The present invention has high reliability and interpretability: The deep detection module combines the clustering analysis and rule verification of large models to improve the reliability of anomaly detection; At the same time, through anomaly credibility scoring and feature deviation description, an interpretability analysis can be performed on the detection results, providing users with a more intuitive basis for anomaly identification and reducing the risk of misjudgment. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The present invention will be further described below with reference to the accompanying drawings.

[0063] Attached Figure 1 is a structural block diagram of a data anomaly detection system based on large and small models. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] The data anomaly detection method, system, device, and medium of the present invention based on large and small models will be described in detail below with reference to the specification drawings and specific embodiments.

[0065] Example 1:

[0066] This embodiment provides a data anomaly detection method based on large and small models. The method first uses an improved Isolation Forest small model for rapid screening, and through a dynamic threshold adjustment mechanism, adaptive feature weight allocation, and multi-scale partitioning optimization, it quickly detects potential anomalies in large-scale data and generates anomaly scores to preliminarily screen suspected anomaly data. Then, it uses a large model to finely detect the suspected anomaly data. Through deep semantic embedding analysis, anomaly feature clustering, and domain rule verification, it accurately identifies the anomaly types and generates highly credible anomaly results. And it classifies and labels the finally detected anomaly data, and adopts cleaning strategies such as deletion, correction, or marking in combination with data characteristics for processing. At the same time, through a feedback optimization mechanism, it further dynamically adjusts the parameters of the small model and the detection strategy of the large model to form an efficient and collaborative anomaly detection and cleaning closed-loop. Specifically as follows:

[0067] S1. Data preprocessing and feature generation: Clean, normalize, and extract features from the input data to obtain preprocessed data, ensuring the quality of the input data and the accuracy of feature expression. Among them, feature extraction includes extracting explicit features through domain knowledge rules and generating implicit features through autoencoders.

[0068] S2. Rapid screening: Through an improved Isolation Forest algorithm, combined with dynamic threshold adjustment, multi-scale partitioning, and feature weight allocation techniques, preliminarily screen the preprocessed data for anomalies and quickly mark suspected anomaly data.

[0069] S3. Deep detection: Finely detect the screened suspected anomaly data, and use a large model of a pre-trained language model or a deep embedding model for semantic embedding analysis, anomaly feature clustering, and rule verification to comprehensively identify anomaly data under complex patterns and classify the anomalies.

[0070] S4. Anomaly data cleaning: Combining the efficient marking ability of the small model and the semantic analysis ability of the large model, adopt cleaning strategies of deletion, correction, or annotation for noise data, potential error data, and out-of-domain data respectively, and dynamically adjust to adapt to multi-scenario requirements to ensure data quality.

[0071] S5. Feedback optimization and closed-loop adjustment: Dynamically adjust the parameters and strategies of each stage, and continuously improve the performance and enhance the adaptability in diverse scenarios by real-time monitoring the detection and cleaning effects. Among them, dynamically adjusting the parameters and strategies of each stage includes optimizing the threshold of rapid screening, the rule base of the large model, and the cleaning strategy.

[0072] The data preprocessing and feature generation in step S1 of this embodiment are specifically as follows:

[0073] S101. Clean the input data through a standardized process. Among them, the cleaning operations include removing redundant information, correcting abnormal formats, and filling in missing values. For noisy data, a strategy combining rule-driven filtering methods and statistical analysis is used. By detecting the skewness, kurtosis, and distribution characteristics of the data, identify data points that may be deviated and perform preliminary correction. At the same time, according to the structural characteristics of the data (such as text, image, table, etc.), corresponding preprocessing methods are adopted. Specifically: for text data, use techniques such as word segmentation, stop word removal, and stemming; for numerical data, perform range normalization or logarithmic transformation to reduce the impact of numerical differences on the model.

[0074] S102. After cleaning the input data, provide multi-dimensional feature expressions by combining explicit feature extraction based on domain rules and implicit feature learning driven by data. Specifically: on the one hand, automatically identify key attributes through domain knowledge and generate explicit features. For example, extract core indicators such as transaction frequency and amount fluctuation in financial data. On the other hand, use autoencoders or deep embedding models to generate implicit features, capturing the deep semantics and potential patterns of the data. The combination of explicit and implicit features ensures the comprehensiveness and effectiveness of data representation.

[0075] S103. Screen the extracted features through a feature importance evaluation mechanism. Specifically: combine statistical methods (such as chi-square test and variance analysis) and model-driven methods (such as gradient-based feature importance scoring) to dynamically adjust the feature set, reducing the interference of redundant features on the subsequent model. At the same time, to support the efficient operation of fast screening, further calculate the weight wi of each feature. The weight of each feature is used to guide the splitting and judgment operations of the small model in the preliminary detection, ensuring the optimization of resource allocation. After data preprocessing and feature generation, the original data is transformed into a high-quality, structured dataset with task-related features, providing a reliable input basis for anomaly detection and improving the efficiency and accuracy of subsequent modules.

[0076] Data preprocessing and feature generation are the foundation. Clean, normalize, structure, and extract features from the input data to provide high-quality data input and accurate feature expressions for subsequent anomaly detection.

[0077] The fast screening in step S2 of this embodiment is based on multiple improvements to the Isolation Forest algorithm, including dynamic threshold adjustment, multi-scale partitioning mechanism, and task-specific feature weight allocation, improving the detection efficiency and sensitivity to complex anomaly patterns. And the improved Isolation Forest algorithm introduces a feature weight-driven multi-scale partitioning mechanism when constructing the splitting tree, specifically as follows:

[0078] S201. When the traditional algorithm randomly selects a single feature for division, it may ignore the influence of key features, resulting in insufficient ability to separate abnormal data. By calculating the importance weight wi of the features and combining with the dynamic sampling strategy, the features with higher weights are given priority in splitting to ensure the contribution of key features to anomaly detection. The calculation formula for weight allocation is as follows:

[0079]

[0080] where Imp(f i ) represents the importance score of feature f i , which is determined based on statistical characteristics (such as variance, information gain) or domain rules; n represents the total number of features;

[0081] S202. Adopt a dynamic threshold adjustment mechanism to dynamically set the anomaly score threshold θ according to the distribution characteristics of the input data (such as mean, standard deviation, and skewness). During the detection process, the Isolation Forest algorithm calculates the path length L(x) for each data point x and generates an anomaly score S(x), and the formula is as follows:

[0082]

[0083] where is the expected value of the path length of normal data;

[0084] S203. Calculate the dynamic threshold θ = μ(S) + α·σ(S) according to the data distribution;

[0085] where μ(S) and σ(S) represent the mean and standard deviation of the anomaly score S respectively; α represents the adjustment parameter for controlling the detection sensitivity. When S(x) ≥ θ, the data point is marked as a suspected anomaly;

[0086] S204. To improve the interpretability of abnormal data, after the screening is completed, the suspected abnormal points are preliminarily labeled, and the distribution deviation of the suspected abnormal points in different feature dimensions is analyzed: by recording the feature dimensions and their path depths at which the abnormal points are separated in the splitting tree, the characteristic description of the abnormal points is generated, providing input information for subsequent in-depth detection.

[0087] Quick screening is the first detection link, aiming to use a lightweight model to quickly screen for anomalies in large-scale data, identify potential abnormal data, and reduce the computational burden of subsequent in-depth detection.

[0088] The in-depth detection in step S3 of this embodiment is specifically as follows:

[0089] S301. Use pre-trained large models such as BERT or GPT to perform semantic embedding analysis on each piece of suspected abnormal data, convert the data into high-dimensional semantic vectors, which can capture the context correlation and deep features of the data; by comparing the similarity with the normal data distribution, evaluate the deviation degree of the data points, and initially judge the possibility of abnormality; at the same time, by analyzing the most significant deviation dimensions in the semantic vectors, identify potential patterns of abnormality, such as abnormal behaviors or distribution abnormalities in specific fields.

[0090] S302. After completing the embedding analysis, perform feature clustering on the suspected abnormal data: based on the distribution density of the embedding vectors, use density clustering algorithms (such as DBSCAN) to group the abnormal data to identify abnormal patterns in the data; for example, in financial transaction data, specific patterns such as "high-frequency trading anomaly" and "sudden increase in amount anomaly" may be identified. The clustering results not only provide a basis for the cleaning module but also enhance the interpretability of the detection results.

[0091] S303. Combine domain rules to verify the abnormal data to further improve the reliability of the detection; the rule library consists of industry-specific abnormal patterns defined by experts. For example, identify abnormal diagnostic results in medical data and detect logical errors in legal documents. By comparing the detection results with the rule library, it is possible to verify whether the abnormal data conforms to known patterns and provide clear judgment criteria for the model.

[0092] S304. To ensure the comprehensiveness and accuracy of the detection, an abnormal credibility scoring mechanism is also adopted to comprehensively evaluate the abnormal nature of each piece of data. The score is generated based on the semantic deviation degree, clustering results, and rule matching scores. Prioritize the processing of abnormal data with high confidence, and leave low-confidence data for subsequent manual review or supplementary detection.

[0093] S305. By combining semantic embedding, clustering analysis, and rule verification, give full play to the advantages of large models in identifying complex abnormal patterns, provide multi-level judgments and classifications for suspected abnormal data, provide high-quality input data for the subsequent cleaning module, and at the same time enhance the detection ability in complex data scenarios.

[0094] The main task of in-depth detection is to further finely detect the data quickly screened and marked as suspected abnormal, make full use of the deep feature extraction and semantic understanding capabilities of the pre-trained large model, accurately judge whether the data is abnormal from multiple dimensions and classify the abnormality; the in-depth detection process realizes high-precision abnormal detection through semantic embedding analysis, abnormal feature clustering, and domain rule verification, making up for the deficiencies of the quick screening module in complex patterns.

[0095] The improved Isolation Forest not only significantly improves the speed of anomaly screening but also enhances the sensitivity to anomaly patterns in multi-dimensional complex data through dynamic optimization, providing high-quality candidate data input for subsequent in-depth detection. By introducing weight assignment and dynamic threshold adjustment, this module can efficiently adapt to different types of input data and provide a robust detection ability for the overall system.

[0096] The anomaly data cleaning in step S4 of this embodiment is specifically as follows:

[0097] S401. Classify and label the anomaly data marked by rapid screening and in-depth detection, and divide the anomaly data into three categories: noise data, potential error data, and out-of-domain data; among them, noise data includes obviously invalid records, and obviously invalid records include missing values, incomplete fields, or meaningless characters; potential error data refers to data points that may deviate from the normal range due to input errors or system anomalies, including large values with abnormal transaction amounts or mutation points in time series; out-of-domain data is data that does not conform to the current task background but may have potential analysis value, such as irrelevant information input across industries.

[0098] S402. Develop differentiated cleaning strategies for different categories of anomaly data; among them, for noise data, the small model directly deletes or fills it through rapid scoring; for example, use rule-driven correction methods to interpolate or fill missing values, and eliminate completely invalid data points to quickly complete the processing; the processing of potential error data depends on the in-depth analysis ability of the large model, uses the semantic understanding of the large model to perform context analysis on the data, and combines the rule library to verify the legality of the data; specifically: for abnormal transaction amounts, the large model generates reasonable correction values through context fitting and verifies whether the corrected data conforms to domain rules; for mutation points in time series, use trend prediction and interpolation repair methods to ensure continuity and logic in the time dimension; for out-of-domain data, the large model further analyzes the potential characteristics of out-of-domain data, compares it with the normal data distribution through semantic embedding, extracts key deviation information and generates annotations, and marks the data for subsequent manual or automated analysis to avoid misdeleting information that may have potential value.

[0099] S403. Design a dynamic cleaning mechanism to adjust the strategy in real time according to the data quality after cleaning and user feedback; for example, if a certain type of noise data appears frequently, optimize the rule-driven cleaning process to improve processing efficiency; if valuable patterns are found in out-of-domain data, record the characteristic deviations and preferentially retain similar data in subsequent cleaning. The dynamic adjustment ability enables the system to adapt to diverse scenario requirements.

[0100] S404. After the cleaning is completed, verify the consistency and integrity of the cleaning results to ensure that the quality of the corrected dataset meets the standards. Re-integrate the data after anomaly handling with the normal data to form a high-quality output dataset; and generate a cleaning report, which details the anomaly classification, correction or marking operations, providing a transparent cleaning process for users. Among them, the cleaning report includes the classification statistics of anomaly detection, the records of cleaning operations, and performance evaluation indicators; the detailed information recorded in the report covers the data quality and context relevance after cleaning for users to refer to or verify.

[0101] Anomaly data cleaning is the core link responsible for processing marked anomaly data. It aims to achieve precise cleaning and classification correction of different types of anomaly data by combining the efficient marking ability of small models and the in-depth understanding ability of large models. The key task of anomaly data cleaning is to ensure high-quality and consistent data after cleaning and provide reliable support for subsequent data usage.

[0102] By combining the fast processing ability of small models and the in-depth analysis advantages of large models, the anomaly data cleaning module has realized a flexible and efficient cleaning strategy, which can not only quickly process simple anomalies but also deeply analyze complex anomaly patterns. Its dynamic adjustment ability enhances the module's adaptability to scenarios, enabling it to provide high-quality data cleaning services in fields such as financial risk control, medical data processing, and industrial monitoring, providing strong guarantees for the accuracy and reliability of downstream tasks.

[0103] The feedback optimization and closed-loop regulation in step S5 of this embodiment are specifically as follows:

[0104] Collect feedback data at each stage, covering the screening accuracy of the fast screening module, the classification accuracy of in-depth detection, and the effect evaluation of cleaning. The specific indicators include the missed detection rate and false detection rate of fast screening, the accuracy of anomaly type classification by in-depth detection, the integrity, consistency, and context relevance of the data after cleaning.

[0105] Feedback optimization and closed-loop regulation are adaptive core links. Through real-time analysis of the detection and cleaning results at each stage, dynamically adjust the parameters and strategies of each module to form an efficient closed-loop processing flow from input data to cleaning completion. The goal of feedback optimization and closed-loop regulation is to improve the overall performance of the system, enhance the adaptability of each module in diverse scenarios, and at the same time achieve automatic optimization and self-learning capabilities.

[0106] Embodiment 2:

[0107] As shown in the Figure 1 appendix, this embodiment provides a data anomaly detection system based on large and small models, and the system includes:

[0108] A data preprocessing and feature generation module, which is used to clean, normalize, and extract features from the input data to generate explicit features and implicit features; among them, the explicit features are extracted based on domain rules, and the implicit features are generated through an autoencoder or a deep embedding model;

[0109] A quick screening module, which is used to preliminarily screen the data based on an improved Isolation Forest algorithm, and improve the ability to quickly identify abnormal data through a dynamic threshold adjustment, multi-scale partitioning, and feature weight allocation mechanism;

[0110] A deep detection module, which is used to conduct refined analysis on the suspected abnormal data screened out by combining with a large model, generate a high-dimensional feature representation through semantic embedding, and achieve abnormal classification through clustering analysis and rule verification;

[0111] An abnormal data cleaning module, which is used to adopt a classification cleaning strategy for noise data, potential error data, and out-of-domain data based on the efficient processing of a small model and the deep semantic analysis of a large model; among them, the classification cleaning strategy includes deletion, correction, and annotation;

[0112] A feedback optimization and closed-loop regulation module, which is used to dynamically adjust the parameters and strategies of each stage to form a closed-loop optimization process from abnormal detection to cleaning.

[0113] In this embodiment, the data preprocessing and feature generation module combines a rule-driven cleaning method with statistical analysis to correct or eliminate noise data; among them, the feature generation process combines statistical methods and model-driven methods to dynamically adjust the feature set and optimize the weight allocation of explicit features and implicit features to support the efficient operation of the quick screening module.

[0114] In this embodiment, the quick screening module is based on an improved Isolation Forest algorithm, optimizes the data splitting strategy with the support of multi-scale partitioning by dynamically adjusting the threshold and combining the feature weight allocation mechanism; at the same time, dynamically sets the abnormal score threshold according to the data distribution to ensure the accuracy and resource allocation efficiency of quick screening, and generates a description of the characteristics of abnormal points for subsequent deep detection.

[0115] In this embodiment, the deep detection module generates high-dimensional semantic embeddings through a large model to capture the deep semantic features of abnormal data, and classifies the abnormal data by combining the density clustering algorithm and domain rule verification; and prioritizes the detection results through an abnormal credibility score, thereby improving the accuracy and interpretability of abnormal classification.

[0116] In this embodiment, the abnormal data cleaning module adopts a differentiated cleaning strategy for different types of abnormal data: directly correcting or removing noise data, generating correction values through large model analysis for potential error data and verifying their legality, and performing semantic deviation analysis on out-of-domain data and generating annotation marks, so as to ensure the quality and integrity of the data after cleaning.

[0117] In this embodiment, the feedback optimization and closed-loop regulation module generates feedback signals by collecting the performance indicators of the fast screening module, the in-depth detection module, and the cleaning module, dynamically adjusts the parameters and strategies of each module; and expands or optimizes the rule base for misjudged abnormal types, supports lightweight fine-tuning of the large model, and finally forms a closed-loop optimization process adapted to different data distributions and domain scenarios.

[0118] In the fast screening module, the feedback mechanism focuses on optimizing the feature weight allocation and detection threshold. Through the analysis of feature importance and screening results, the module dynamically adjusts the priority of features. For example, it increases the weight of features that perform outstandingly in multiple detections to ensure more accurate subsequent screening. In addition, when there is a significant deviation between the results of fast screening and the judgment of the in-depth detection module, the feedback signal triggers an adjustment of the screening threshold to adapt to the characteristics of the new data distribution and improve the screening efficiency.

[0119] For the in-depth detection module, the feedback mechanism optimizes the detection strategy and rule base of the large model by monitoring the accuracy of classification results and rule verification. If certain abnormal patterns are misjudged multiple times in detection, the system will expand or adjust the rule base and optimize the semantic embedding algorithm of the large model to make it more accurately capture the abnormal features of specific scenarios. In addition, the module supports lightweight fine-tuning of specific parameters of the large model to improve its performance in domain abnormal classification.

[0120] The cleaning module optimizes the cleaning strategy according to the feedback signal. For example, when there are still quality problems (such as insufficient consistency or excessive correction error) in a certain type of abnormal data after cleaning, the module will reselect or combine cleaning methods, adjust the matching range or correction logic of the rules. At the same time, the direct feedback from users will also be incorporated into the system, and the cleaning process will be gradually improved through interactive adjustment of strategies to ensure meeting diverse requirements.

[0121] Finally, the feedback optimization module forms an efficient closed-loop regulation mechanism by integrating these dynamic adjustment results into the system. The system continuously learns new data distributions and abnormal patterns, gradually improves the performance of detection and cleaning, and adapts to the requirements of different domains and scenarios. This module can not only significantly improve the accuracy of detection and cleaning, but also enhance the robustness and adaptability of the system, providing a solid guarantee for dealing with complex and diverse data scenarios.

[0122] Embodiment 3:

[0123] This embodiment also provides an electronic device, including: a memory and a processor;

[0124] Wherein, the memory stores computer-executable instructions;

[0125] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the data anomaly detection method based on the size model in any embodiment of the present invention.

[0126] The processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0127] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory may also include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash memory card, at least one magnetic disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0128] Embodiment 4:

[0129] This embodiment also provides a computer-readable storage medium for the data anomaly detection method based on the size model. Specifically, a system or device equipped with a storage medium can be provided. Software program codes for implementing the functions in any one of the above embodiments are stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0130] In this case, the program code read from the storage medium itself can implement the functions in any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0131] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0132] In addition, it should be clear that not only can the actual operations be completed in part or in whole by executing the program code read by the computer, but also by instructing the operating system and the like operating on the computer based on the program code, thereby implementing the functions of any one of the above embodiments.

[0133] In addition, it can be understood that the program code read from the storage medium is written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then based on the instructions of the program code, the CPU and the like installed on the expansion board or the expansion unit execute part or all of the actual operations, thereby implementing the functions of any one of the above embodiments.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data anomaly detection method based on a size model, characterized in that: The method first uses the improved Isolation Forest small model for rapid screening, and through the dynamic threshold adjustment mechanism, adaptive feature weight allocation and multi-scale partition optimization to quickly detect potential anomalies in large-scale data and generate anomaly scores, and preliminarily screen suspected abnormal data; then use the large model to perform fine detection on suspected abnormal data, and accurately identify the anomaly type and generate highly reliable abnormal results through deep semantic embedding analysis, abnormal feature clustering and domain rule verification; and finally classify and annotate the abnormal data detected, and use the cleaning strategy of deletion, correction or marking to process it according to the data characteristics, and further dynamically adjust the small model parameters and the large model detection strategy through the feedback optimization mechanism to form an efficient and coordinated anomaly detection and cleaning closed loop; the details are as follows: Data preprocessing and feature generation: clean, normalize and extract features from input data to obtain preprocessed data to ensure the quality of input data and the accuracy of feature expression; feature extraction includes extracting explicit features through domain knowledge rules and generating invisible features through autoencoders; Fast screening: The improved Isolation Forest algorithm is combined with dynamic threshold adjustment, multi-scale partitioning and feature weight allocation technology to perform preliminary anomaly screening on the pre-processed data and quickly mark suspected abnormal data; Deep detection: Perform detailed detection on the screened suspected abnormal data, use the pre-trained language model or the large model of the deep embedding model to perform semantic embedding analysis, abnormal feature clustering and rule verification, comprehensively identify abnormal data in complex patterns and perform abnormal classification; Abnormal data cleaning: Combining the efficient labeling capability of the small model with the semantic analysis capability of the large model, we adopt cleaning strategies of deletion, correction or annotation for noise data, potential error data and out-of-domain data, and dynamically adjust to meet the needs of multiple scenarios; Feedback optimization and closed-loop regulation: Dynamically adjust the parameters and strategies of each stage, and continuously improve performance through real-time monitoring of detection and cleaning effects; dynamic adjustment of parameters and strategies of each stage includes optimizing the threshold of rapid screening, the rule base of large models, and cleaning strategies.

2. The data anomaly detection method based on size model according to claim 1, characterized in that: Data preprocessing and feature generation are as follows: The input data is cleaned through a standardized process; the cleaning operation includes removing redundant information, correcting abnormal formats, and filling missing values; for noisy data, a strategy combining rule-driven filtering methods and statistical analysis is used to detect the skewness, kurtosis and distribution characteristics of the data, identify data points that may have deviations and make preliminary corrections; at the same time, the corresponding preprocessing methods are adopted according to the structural characteristics of the data, specifically: for text data, word segmentation, stop word removal and stem extraction techniques are adopted; for numerical data, range normalization or logarithmic transformation is performed to reduce the impact of numerical differences on the model; after the input data is cleaned, a combination of explicit feature extraction based on domain rules and data-driven implicit feature learning is used to provide multi-dimensional feature expression; specifically: on the one hand, key attributes are automatically identified and explicit features are generated through domain knowledge; on the other hand, implicit features are generated using autoencoders or deep embedding models to capture the deep semantics and potential patterns of the data; The extracted features are screened through the feature importance evaluation mechanism; specifically: the feature set is dynamically adjusted by combining statistical methods and model-driven methods to reduce the interference of redundant features on subsequent models; at the same time, the weight w of each feature is further calculated i ,The weight of each feature is used to guide the splitting and judgment operations of the small model in preliminary detection.

3. The data anomaly detection method based on size model according to claim 1, characterized in that: Fast screening is based on the Isolation Forest algorithm with multiple improvements on dynamic threshold adjustment, multi-scale partitioning mechanism, and task-specific feature weight allocation. The improved Isolation Forest algorithm introduces a feature weight-driven multi-scale partitioning mechanism when constructing the split tree, as follows: By calculating the importance weight w of the feature i , combined with the dynamic sampling strategy, the features with higher weights are given priority during splitting to ensure the contribution of key features to anomaly detection; the weight distribution calculation formula is as follows: Among them, Imp(f i ) represents the feature f i The importance score is determined based on statistical characteristics or domain rules; n represents the total number of features; A dynamic threshold adjustment mechanism is adopted to dynamically set the anomaly score threshold θ according to the distribution characteristics of the input data. During the detection process, the Isolation Forest algorithm calculates the path length L(x) for each data point x and generates an anomaly score S(x), as shown in the following formula: in, is the expected value of normal data path length; Calculate the dynamic threshold θ = μ(S) + α·σ(S) according to the data distribution; Where μ(S) and σ(S) represent the mean and standard deviation of the anomaly score S, respectively; α represents the adjustment parameter that controls the detection sensitivity; when S(x) ≥ θ, the data point is marked as a suspected anomaly; After the screening is completed, the suspected outliers are preliminarily marked, and the distribution deviation of the suspected outliers in different feature dimensions is analyzed: by recording the feature dimensions and path depths of the outliers separated in the split tree, a characteristic description of the outliers is generated to provide input information for subsequent deep detection.

4. The data anomaly detection method based on size model according to claim 1, characterized in that: The depth detection is as follows: Use BERT or GPT pre-trained large models to perform semantic embedding analysis on each piece of suspected abnormal data, converting the data into a high-dimensional semantic vector that can capture the contextual associations and deep features of the data; by comparing the similarity with the normal data distribution, evaluate the degree of deviation of the data point and preliminarily determine the possibility of abnormality; After completing the embedding analysis, feature clustering is performed on the suspected abnormal data: based on the distribution density of the embedding vector, a density clustering algorithm is used to group the abnormal data to identify the abnormal patterns of the data; Verify abnormal data in combination with domain rules to further improve detection reliability; The rule base consists of industry-specific anomaly patterns defined by experts. By comparing the detection results with the rule base, it is possible to verify whether the anomaly data conforms to the known pattern and provide clear judgment criteria for the model; An abnormal credibility scoring mechanism is also used to comprehensively evaluate the abnormal nature of each data. The score is generated based on the degree of semantic deviation, clustering results, and rule matching scores. High-confidence abnormal data is processed first, and low-confidence data is reserved for subsequent manual review or supplementary testing; By combining semantic embedding, cluster analysis and rule verification, the advantages of large models in complex abnormal pattern recognition are fully utilized to provide multi-level judgment and classification for suspected abnormal data, provide high-quality input data for subsequent cleaning modules, and improve detection capabilities in complex data scenarios.

5. The data anomaly detection method based on size model according to claim 1, characterized in that: The specific steps for cleaning abnormal data are as follows: The abnormal data that have been quickly screened and deeply detected are classified and labeled, and the abnormal data are divided into three categories: noise data, potential error data, and out-of-domain data; among them, noise data includes obviously invalid records, and obviously invalid records include missing values, incomplete fields, or meaningless characters; potential error data refers to data points that deviate from the normal range due to input errors or system anomalies; out-of-domain data is data that does not match the current task context but may have potential analysis value; Differentiated cleaning strategies are formulated for different categories of abnormal data. For noise data, the small model directly deletes or fills it through rapid scoring. The processing of potentially erroneous data depends on the deep analysis capabilities of the large model, which uses the semantic understanding of the large model to conduct contextual analysis of the data and verify the legitimacy of the data in combination with the rule base. Specifically, for abnormal transaction amounts, the large model generates reasonable correction values ​​through context fitting and verifies whether the corrected data complies with domain rules. For mutation points in time series, trend prediction and interpolation repair methods are used to ensure continuity and logic in the time dimension. For out-of-domain data, the large model further analyzes the potential characteristics of out-of-domain data, compares it with normal data distribution through semantic embedding, extracts key deviation information and generates annotations, and marks data for subsequent manual or automated analysis to avoid accidental deletion of information that may have potential value. Design a dynamic cleaning mechanism to adjust the strategy in real time based on the quality of cleaned data and user feedback; After the cleaning is completed, the consistency and integrity of the cleaning results are verified to ensure that the quality of the corrected data set meets the standards. The data after anomaly processing is reintegrated with the normal data to form a high-quality output data set. A cleaning report is generated to record the anomaly classification, correction or marking operations in detail to provide users with a transparent cleaning process. The cleaning report includes classification statistics of anomaly detection, records of cleaning operations and performance evaluation indicators. The detailed information recorded in the report covers the quality and contextual relevance of the cleaned data for user reference or verification.

6. The data anomaly detection method based on size model according to any one of claims 1 to 5, characterized in that: Feedback optimization and closed-loop regulation are as follows: Feedback data from each stage is collected, covering the screening accuracy of the rapid screening module, the classification accuracy of the deep detection, and the evaluation of the cleaning effect. Specific indicators include the missed detection rate and false detection rate of rapid screening, the accuracy of deep detection in classifying abnormal types, and the integrity, consistency, and contextual relevance of the cleaned data.

7. A data anomaly detection system based on a size model, characterized in that: The system includes: Data preprocessing and feature generation module, which is used to clean, normalize and extract features from input data to generate explicit features and implicit features. Explicit features are extracted based on domain rules, and implicit features are generated through autoencoders or deep embedding models. The fast screening module is used to perform preliminary screening of data based on the improved Isolation Forest algorithm. It improves the ability to quickly identify abnormal data through dynamic threshold adjustment, multi-scale division and feature weight allocation mechanism; The deep detection module is used to conduct refined analysis on the screened suspected abnormal data in combination with the large model, generate high-dimensional feature representation through semantic embedding, and realize abnormal classification through clustering analysis and rule verification; The abnormal data cleaning module is used for efficient processing based on small models and deep semantic analysis of large models. It adopts classification cleaning strategies for noisy data, potential error data, and out-of-domain data. Among them, the classification cleaning strategies include deletion, correction, and annotation. The feedback optimization and closed-loop regulation module is used to dynamically adjust the parameters and strategies of each stage to form a closed-loop optimization process from anomaly detection to cleaning.

8. The data anomaly detection system based on size model according to claim 7, characterized in that: The data preprocessing and feature generation module corrects or removes noise data by combining rule-driven cleaning methods and statistical analysis; wherein the feature generation process combines statistical methods and model-driven methods to dynamically adjust the feature set and optimize the weight distribution of explicit features and implicit features to support the efficient operation of the rapid screening module; The rapid screening module is based on the improved Isolation Forest algorithm. It optimizes the data splitting strategy with the support of multi-scale partitioning by dynamically adjusting the threshold and combining the feature weight allocation mechanism. At the same time, it dynamically sets the anomaly scoring threshold according to the data distribution to ensure the accuracy of rapid screening and the efficiency of resource allocation, and generates a description of the anomaly point characteristics for subsequent in-depth detection. The deep detection module generates high-dimensional semantic embedding through a large model, captures the deep semantic features of abnormal data, and classifies abnormal data by combining density clustering algorithm and domain rule verification; and prioritizes the detection results through abnormal credibility scoring, thereby improving the accuracy and interpretability of abnormal classification; The abnormal data cleaning module adopts differentiated cleaning strategies for different types of abnormal data: directly correct or eliminate noise data, generate correction values ​​and verify the legitimacy of potential error data through large model analysis, and perform semantic deviation analysis on out-of-domain data and generate annotation tags to ensure the quality and integrity of the cleaned data; The feedback optimization and closed-loop adjustment module generates feedback signals by collecting the performance indicators of the rapid screening module, deep detection module and cleaning module, and dynamically adjusts the parameters and strategies of each module; it also expands or optimizes the rule base for misjudged anomaly types, supports lightweight fine-tuning of large models, and ultimately forms a closed-loop optimization process that adapts to different data distributions and domain scenarios.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor performs the size model-based data anomaly detection method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the size model-based data anomaly detection method according to any one of claims 1 to 6.

Images