Table data anomaly detection method and system based on genetic algorithm learning normal form

Abstract

The invention provides a table data anomaly detection method and system based on a genetic algorithm learning normal form, and belongs to the technical field of abnormal data detection.The method comprises the steps that firstly, a dirty data set to be detected is obtained, an initial code is obtained in combination with a word vector model and a TF-IDF algorithm, weighting is carried out by means of a context attention mechanism, and data instance code representation is obtained; then, a genetic algorithm serves as a data selector, a data set is divided into tuple individuals, the individual fitness is calculated through a comprehensive fitness function, and population initialization and selection strategy design are completed; then performing crossover and mutation operations, screening population diversity individuals, and preliminarily labeling high-probability normal and abnormal individuals; and finally, on the basis of the labeled sample set, positive and negative sample packets are established by a label diffusion mechanism, pseudo-label expansion training data are generated through adaptive iteration, and comprehensive judgment of table data exception is realized through model training. According to the invention, the problem that diversified errors cannot be comprehensively and systematically covered without human participation in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of anomaly detection technology, specifically to a method and system for detecting tabular data anomalies based on a genetic algorithm learning paradigm. Background Technology

[0002] Anomaly detection, as a core technology for data quality assurance, plays a crucial role in data-driven decision-making. It aims to identify various quality issues in data, including anomalies such as erroneous values, missing values, duplicate records, and inconsistencies. These data quality issues can stem from various factors, including system failures, human input errors, or defects in the data acquisition process. Failure to detect and address them promptly will severely impact the reliability of downstream applications and the effectiveness of decision-making. Therefore, building an efficient and accurate anomaly detection mechanism has become a critical prerequisite for ensuring the successful deployment of data-driven applications.

[0003] Traditional anomaly detection methods have many limitations. First, most methods only target specific types of anomalies in the data, such as numerical anomalies or categorical anomalies, making it difficult to adapt to complex and diverse real-world data environments. Second, many methods rely heavily on the prior knowledge of domain experts, requiring manual definition of detection rules or setting of threshold parameters, which not only consumes a lot of manpower and time but also limits the scalability and generality of the methods. In recent years, the rapid development of machine learning, especially deep learning technology, has brought new opportunities for tabular data anomaly detection. Researchers have proposed a variety of learning-based methods, attempting to reduce the workload required for manual intervention by automatically learning data patterns. However, these methods face new challenges: (1) supervised learning models rely on a large amount of high-quality labeled data, and the cost of obtaining these labels is extremely high; (2) in practical applications, anomalous samples are usually far fewer than normal samples, leading to serious class imbalance problems. For some key areas, such as rare disease diagnosis or financial fraud detection, anomalous data itself is extremely scarce, making model training extremely difficult.

[0004] Traditional anomaly detection algorithms focus only on specific types of errors, and can be broadly categorized into quantitative and qualitative error detection based on the nature of the errors. Quantitative data error detection aims to identify outliers exceeding frequency or distance thresholds; for example, the FAHE framework specifically detects (pseudo)missing values, while the DBoost comprehensive framework integrates multiple outlier detection strategies to identify data errors. Qualitative data error detection primarily focuses on violations of data integrity or predefined patterns. Methods such as GARF, CDC, and MLNClean identify non-compliant data by applying integrity constraints. Among these, HoloClean and NADEEF have constructed rule-based detection methods that address rule violations by enforcing functional dependencies and integrity constraints. However, these methods heavily rely on the expertise of data specialists, leading to complex system configurations and their effectiveness largely dependent on the quality of manually provided rules. In contrast, the detection method proposed in this study can comprehensively identify different types of data anomalies, significantly expanding the applicability of traditional methods.

[0005] Automated strategies using machine learning can reduce reliance on external corpora and human intervention, and capture various types of errors. However, this learning method heavily depends on high-quality datasets with uniform distribution. Obtaining training or historical datasets is very difficult in some specialized domains, creating a bottleneck in the learning task. SAGED uses automated strategies to reduce reliance on external corpora and human intervention, but requires a small number of historical training samples for model training. Tools such as RAHA and Activeclean simplify the data cleaning process, but still require human intervention to ensure the feasibility and accuracy of the model. While existing anomaly detection technologies have made some progress in identifying errors in tabular data, they still face insurmountable technical bottlenecks when dealing with complex and changing data environments. This leads to models failing to operate stably without human intervention, and even detection failures. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a tabular data anomaly detection method and system based on a genetic algorithm learning paradigm. This method overcomes the limitations of existing tabular data anomaly detection methods, namely: some methods are often limited to specific data types and are difficult to apply effectively in general scenarios. While machine learning-based methods can cover multiple types of errors, they heavily rely on large amounts of high-quality labeled data, leading to high system costs. The introduction of few-shot learning techniques alleviates the labeling burden to some extent, but the uneven distribution of labeled and unlabeled data still significantly restricts the improvement of model performance.

[0007] This invention is achieved through the following technical solution: a method for detecting anomalies in tabular data based on a genetic algorithm learning paradigm. The method specifically includes the following steps: Step 1: Obtain the dirty dataset to be detected, combine the word vector model and TF-IDF algorithm to obtain the initial encoding form of the data, and use the synergistic effect of the context attention mechanism to complete the encoding weighting to obtain the encoded representation of the data instance; Step 2: Construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection; Step 3: Perform crossover and mutation operations on the individuals in the population divided in Step 2, and introduce a population diversity fusion strategy to screen individuals, and finally complete the labeling of high-probability normal individuals and abnormal individuals. Step 4: Based on the sample set labeled in Step 3, dynamically construct positive and negative class sample bags by improving the label diffusion mechanism, expand the training data during the adaptive iterative generation of pseudo labels, and execute model training. Step 5: Process the tabular data to be detected using the model trained in Step 4; the model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data according to a preset threshold, and finally outputs the anomaly detection result.

[0008] Furthermore, step 1 specifically includes: Step 1.1: Obtain the dirty dataset of the table. For each cell instance, use Word2vec to calculate the word vector and use the TF-IDF algorithm to obtain the term frequency-inverse file encoding. Step 1.2: Concatenate the word vectors and word frequency codes, and then weight them according to the context attention mechanism to obtain the final encoded vector.

[0009] Furthermore, step 2 specifically involves: Step 2.1: Decompose the dataset into sets of tuples by row, and treat each tuple as an independent individual in the genetic algorithm; Step 2.2: Randomly select a subset of k tuples from the set of tuples, and use a comprehensive fitness function to evaluate the degree of abnormality of each cell; The comprehensive fitness function integrates three key anomaly detection components, including a function dependency violation anomaly score based on the TANE algorithm, a statistical outlier detection score, and an adaptive mode anomaly detection score. Step 2.3: Use the TANE algorithm-based function dependency violation detection method to obtain attribute partitions and value partitions, and label the instance labels in the qualitative data according to the final generated rules; Step 2.4: Use statistical-based quantitative data anomaly detection, combining Gaussian distribution and histogram, to generate outlier labels for the quantitative data portion; Step 2.5: Use a self-learning pattern detection method to dynamically generate a set of regular expression rules, establish a column-level dominant pattern representation, and divide it into dominant pattern, secondary pattern and noise pattern, thereby labeling abnormal tags that do not conform to the expected format. Step 2.6: Based on the fitness in Step 2.5, complete the population initialization and individual selection.

[0010] Furthermore, step 3 specifically involves: Step 3.1: Perform a crossover operation on the randomly selected tuple individuals in Step 2.2. Exchange some contents of the paired individuals according to their fitness differences, prioritizing the exchange of cells with larger fitness differences, and generate new tuples. Randomly select tuples with high fitness for mutation operation, and randomly select new values ​​from candidate cells in the same column to replace the mutated tuples. Step 3.2: By analyzing the distribution characteristics of cells in each column within the population, the selection weights are dynamically adjusted to perform diversity fusion selection of the population. The average fitness of all cells is calculated as the baseline threshold to obtain high-fitness and low-fitness populations. Individuals with a high probability of being normal or abnormal are labeled.

[0011] Furthermore, step 4 specifically involves: Step 4.1: Based on the labeled sample set in Step 3.2, the sample features and label labels are obtained. The cosine similarity metric is used to calculate the distance between unlabeled samples and labeled samples in the column level. For unlabeled samples, they are selected and assigned to the set of labels of the labeled samples with the highest cosine similarity. Step 4.2: Divide the set into groups. Each positive package consists of a fixed proportion of positive instances in the column and a random number of global positive instances. Each negative package consists of a fixed proportion of negative instances in the column and a random number of global negative instances. Generate a negative package-level instance set. Step 4.3: Reconstruct the negative packet set using an autoencoder, minimize the reconstruction error, and calculate the loss function to obtain the loss value on the negative packet set; select the negative packet set with the smallest reconstruction error as the most reliable negative packets and input them into the supervised network along with the positive packets as the dataset for training; output the anomalous probability of each instance in each packet and map it to the anomalous probability value of the packet, and use backpropagation to minimize the loss; Step 4.4: In each round, continuously select the most reliable negative and positive packets and input them into the supervised model for training to obtain the optimal parameters. Output the comprehensive anomaly label for all instances to complete the model training.

[0012] A tabular data anomaly detection system based on a genetic algorithm learning paradigm; The system includes an encoding weighting module, a screening module, a crossover and mutation module, a training module, and a prediction module. The encoding weighting module obtains the dirty dataset to be detected, combines the word vector model and the TF-IDF algorithm to obtain the initial encoding form of the data, and uses the synergistic effect of the context attention mechanism to complete the encoding weighting to obtain the encoded representation of the data instance; The screening module is used to construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection. The crossover and mutation module performs crossover and mutation operations on the individuals in the population divided by the screening module, and introduces a population diversity fusion strategy to screen individuals, and finally completes the labeling of high-probability normal individuals and abnormal individuals. The training module is based on the sample set labeled by the cross-mutation module. It dynamically constructs positive and negative class sample bags through an improved label diffusion mechanism, expands the training data during the adaptive iterative generation of pseudo labels, and performs model training. The prediction module processes the tabular data to be detected using the model trained by the training module; the model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data according to a preset threshold, and finally outputs the anomaly detection result.

[0013] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0014] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] Beneficial effects of the invention This invention provides a method for detecting anomalies in tabular data based on a genetic algorithm learning paradigm. It utilizes a genetic algorithm to screen high-quality samples and assign initial labels, reducing reliance on manual annotation and effectively lowering the high system costs associated with annotation requirements. Secondly, an improved label diffusion algorithm is introduced to alleviate the problem of uneven data distribution. Finally, to avoid model performance crashes, an iterative adaptive generation strategy for pseudo-labels is designed to achieve comprehensive anomaly data judgment, solving practical problems encountered in the field of tabular data.

[0016] This invention is applicable to anomaly detection technology in the field of structured data anomaly detection. Attached Figure Description

[0017] Figure 1 This is a flowchart of the tabular data anomaly detection method based on the genetic algorithm learning paradigm described in this invention.

[0018] Figure 2 This is a model diagram of the tabular data anomaly detection method based on the genetic algorithm learning paradigm described in this invention.

[0019] Figure 3 This is an overall flowchart of the data selector based on the heritage algorithm described in this invention.

[0020] Figure 4 This is a flowchart of the data selector selecting tuples according to the present invention.

[0021] Figure 5 This is a schematic diagram of tuple crossover and mutation as described in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Unless otherwise specified, the experimental methods used in the following examples are conventional methods. Unless otherwise specified, the materials, reagents, methods, and instruments used are all conventional materials, reagents, methods, and instruments in the art, and can be obtained commercially by those skilled in the art.

[0024] Combination Figures 1 to 5 This invention first acquires the dirty dataset to be detected, and combines word vector model and TF-IDF algorithm to obtain the initial encoding form of the data. The synergistic effect of context attention mechanism is used to complete the encoding weighting, and obtain the encoded representation of data instances. Genetic algorithm is used as data selector to decompose the dataset into individuals in units of tuples. The fitness score of each individual is calculated by using the constructed comprehensive fitness function to complete the population initialization and selection strategy design. Crossover and mutation operations are performed, and individual fusion screening is performed on the population diversity to complete the preliminary labeling of high-probability normal individuals and abnormal individuals. Finally, based on the labeled sample set, positive and negative sample bags are dynamically constructed through label diffusion mechanism. The training data is expanded in the process of adaptively iteratively generating pseudo-labels, and model training is performed to achieve a comprehensive judgment of tabular data anomalies.

[0025] Implementation method one: such as Figure 1 As shown; this invention proposes a method for detecting anomalies in tabular data based on a genetic algorithm learning paradigm. Step 1: Obtain the dirty dataset to be detected, construct a word vector model based on the data features, and encode the data by combining the TF-IDF algorithm and the context attention mechanism to obtain the encoded representation of the data instances; Step 1.1: Obtain the dirty dataset of the table. For each cell instance, use Word2vec technology to calculate the word vector, and use the TF-IDF algorithm to obtain the term frequency-inverse file encoding. Step 1.2: Concatenate the word vectors and word frequency codes, and then use the context attention mechanism to weight the instance to obtain the final encoded vector.

[0026] Step 2: Construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection; Step 2.1: Decompose the dataset into sets of tuples by row, and treat each tuple as an independent individual in the genetic algorithm; This method is applicable to a given dirty dataset. First, it is decomposed into a set of tuples of length n by row. ,in Let represent the i-th data tuple. Each tuple is considered an independent individual in the genetic algorithm, providing a basis for subsequent evolutionary operations.

[0027] Step 2.2: Randomly select a subset of k tuples from the tuple set and evaluate the anomaly level of each cell using a comprehensive fitness function. This function integrates three key anomaly detection components: a function dependency violation detection score based on the TANE algorithm, a statistical outlier detection score, and an adaptive mode anomaly detection score.

[0028] Design a comprehensive fitness function The function assesses the anomaly level of each cell by fusing three key anomaly detection components: a function dependency violation anomaly score based on the TANE algorithm, a statistical outlier detection score, and an adaptive mode anomaly detection score. Here, α, β, and γ are weighting coefficients that satisfy... .

[0029]

[0030] Step 2.3: Use the TANE algorithm-based function dependency violation detection method to obtain attribute partitions and value partitions, and label the instance labels in the qualitative data according to the final generated rules; Functional dependency violation detection based on the TANE algorithm identifies outliers that violate dependency rules by mining functional dependencies in the data. Starting with a single attribute, it gradually expands to a larger set of attributes, using difference sets to efficiently compute functional dependencies. This ensures that only non-trivial functional dependencies are considered. Simultaneously, pruning rules are used to exclude attribute combinations that are unlikely to generate new functional dependencies.

[0031]

[0032] in, It is the set of functional dependencies involving the j-th column. Determine if a cell violates the dependency function fd. It is an indicator function. For the rule list extracted by the TANE algorithm, the rules are used to detect cells and generate anomaly scores.

[0033] Step 2.4: Use statistical-based quantitative data anomaly detection, combining Gaussian distribution and histogram, to generate outlier labels for the quantitative data portion;

[0034] The histogram method and the Gaussian method are represented by the following formulas, where It is the range of values ​​in the j-th column. and These are the mean and standard deviation of the j-th column, respectively.

[0035]

[0036]

[0037] Step 2.5: Use a self-learning pattern detection method to dynamically generate a set of regular expression rules, establish a column-level dominant pattern representation, and divide it into dominant pattern, secondary pattern and noise pattern, and label abnormal tags that do not conform to the expected format accordingly.

[0038] To address the common format inconsistency issue in mixed data types, an adaptive pattern detection mechanism was designed. This mechanism automatically analyzes data column characteristics, dynamically generates a set of regular expression rules, and establishes a column-level dominant pattern representation. Let the data column... It contains n cells and detects k different pattern categories. The theoretical expected frequency when the pattern is randomly distributed is:

[0039] The deviation of the observed distribution from the theoretical distribution is measured, and the direction of significant deviation for each model is identified using standardized residual calculations.

[0040] in, For pattern The actual observation frequency, the model classification rule is when At this point, the dominant type of the column is the pattern, and a corresponding regular expression rule is generated to mark it as a normal value. At this time, if the column is a secondary type, the pattern is retained but no detection rules are generated. If the time frame indicates noise, it is marked as an anomaly candidate. The pattern anomaly score can be derived from different types:

[0041] Step 2.6: Based on the fitness in Step 2.5, complete the population initialization and individual selection.

[0042] A subset consisting of k randomly selected tuples from the set of tuples T is denoted as . They became a population, divided into several groups. Each population. Each tuple The fitness of a given value is a weighted sum of the fitness values ​​of all its cells:

[0043] A dual-threshold selection strategy was employed, selecting individuals with high and low fitness thresholds to represent individuals with high degrees of abnormality and high degrees of normality, respectively. The high fitness threshold was... The low fitness threshold is ,in, and These are the mean and standard deviation of the current population fitness, respectively.

[0044]

[0045] Step 3: Perform crossover and mutation operations on the individuals in the population divided in Step 2, and introduce a population diversity fusion strategy to screen individuals, and finally complete the labeling of high-probability normal individuals and abnormal individuals. Step 3.1: In Step 2.2, a crossover operation is performed on the selected tuples, swapping some contents of the paired individuals based on their fitness differences. The mutation operation randomly selects tuples with higher fitness, and randomly selects new values ​​from candidate cells in the same column to replace the mutated tuples. Perform a crossover operation on the selected tuples to pair the individuals. and Swap some content based on fitness differences. Prioritize swapping cells with larger fitness differences to generate new tuples. and The mutation operation randomly selects tuples with higher fitness. For the cells in Randomly select new values ​​from candidate cells in the same column to replace them, generating mutated tuples. The mutation probability is set to 0.1 of the total number of transformed tuples.

[0046] Step 3.2: By analyzing the distribution characteristics of cells in each column within the population, the selection weights are dynamically adjusted to perform diversity fusion selection of the population. The average fitness of all cells is calculated as the baseline threshold to obtain high-fitness and low-fitness populations. Individuals with a high probability of being normal or abnormal are labeled.

[0047] For highly fit populations First, calculate the average fitness of all cells as the baseline threshold:

[0048] in This represents the size of the high-fitness population, where n is the dimension of the data attribute. Based on this threshold, the proportion of cells in column j with fitness exceeding the average fitness is calculated as the diversity weight for that column.

[0049] This weight reflects the concentration of column j in high-fitness selection. When When the value is close to 1, it indicates that most cells in the column have high fitness; when... A smaller value indicates that high-fitness cells in that column are relatively scarce. For low-fitness populations... The average fitness of the population is calculated using a dual-method approach:

[0050] Then, the proportion of cells in each column with below-average fitness is calculated as the diversity weight:

[0051] Incorporating diversity weights into fitness calculation yields the fusion selectivity:

[0052] This fusion mechanism can effectively regulate the selection preferences of each column. For high-fitness populations, columns with smaller weights will be given higher selection priority; for low-fitness populations, columns with smaller weights will also receive more attention.

[0053]

[0054] The final selection set is defined as:

[0055] in Includes the cell (y=1) that is marked as an anomaly candidate. Includes cells marked as normal candidates (y=0).

[0056] Step 4: Based on the sample set labeled in Step 3, dynamically construct positive and negative class sample bags through the label diffusion mechanism, expand the training data during the adaptive iterative generation of pseudo-labels, and execute model training.

[0057] Step 4.1: Based on the sample set annotated in Step 3.2, the sample features and label annotations are obtained. The cosine similarity metric is used to calculate the distance between unlabeled samples and labeled samples in the column level. For unlabeled samples, they are selected and assigned to the set of labels of the labeled samples with the highest cosine similarity.

[0058] Let the known set of labels be ,in , indicating normal or abnormal labels; unlabeled sample sets are ,in To be determined. The embedding vectors for all samples are calculated from step 1. Cosine similarity is used to measure the distance between unlabeled samples and all labeled samples:

[0059] For unlabeled samples Select the labeled sample with the highest similarity to the cosine. ,Right now:

[0060] And assign the unlabeled sample to Within the tag cluster, let To further enhance the accuracy of the label diffusion results, a neighborhood consistency verification mechanism is introduced. After the initial label diffusion is completed, a k-nearest neighbor strategy is used to verify the local consistency of each unlabeled sample that has obtained a propagated label.

[0061] By statistically analyzing the label distribution of the target sample's k nearest neighbors, if a significant conflict is found between its current label and the neighboring labels, the sample is marked as an uncertain sample and removed from the current training set, thereby preventing label contamination.

[0062] Step 4.2: Divide the set into groups. Each positive package consists of a fixed proportion of positive instances in the column and a random number of global positive instances. Each negative package consists of a fixed proportion of negative instances in the column and a random number of global negative instances. Generate a negative package-level instance set. Introducing the concept of multi-instance learning, we construct a package space from k instances and provide label settings only at the package level, satisfying the following conditions: positive package At least one Examples in ( The package tag can be obtained. Negative package All instances belong to instances in U, and this package tag can be represented as .

[0063] By constructing the package probability function F and the instance probability function f, the model reasonably assigns an abnormal label to each instance and constructs a mapping function. When y=1, it indicates that the cell is ultimately abnormal, and when y=0, it indicates that the instance is ultimately normal, thus solving the problem of incomplete labeling.

[0064] A set of positive and negative packets is constructed using a dynamic sampling method. K×θ core positive instances are extracted from all cells labeled as "abnormal" in the i-th column, and (1-θ)×K positive samples are randomly selected from all globally abnormal cells. This process iteratively generates a sufficient number of positive packets. For negative packets It consists of instances labeled as normal in the i-th column, using the same... Using the same method, K×θ core negative instances are selected from the local columns, and (1-θ)×K negative samples are randomly drawn from the global normal cells to construct the negative package.

[0065] Step 4.3: Reconstruct the negative packet set using an autoencoder, minimize the reconstruction error, and calculate the loss function to obtain the loss value on the negative packet set; select the negative packet set with the smallest reconstruction error as the most reliable negative packets and input them into the supervised network along with the positive packets as the dataset for training; output the anomalous probability of each instance in each packet and map it to the anomalous probability value of the packet, and use backpropagation to minimize the loss; The input instances are mapped to a lower-dimensional hidden space and then back to the original space. A higher reconstruction error indicates a more significant difference from normal data and a higher probability of anomalies. The method for determining the reconstruction error is as follows: , It is the Euclidean norm. It is the reconstructed input. This represents the network's weights and other parameters. Design and train an encoder with a two-component loss function to detect anomalous states in the reconstructed data. , This represents a simulation of the distribution of observed instances on unlabeled data. This means that positive and negative instance labels are learned from the provided positive and negative example packages, and finally the loss is minimized using common backpropagation techniques.

[0066] Reconstructing unlabeled packet loss Using an autoencoder, any form of exception can be captured; its reconstruction error for negative example packets can be expressed as... The unmarked portion of the total It can be represented as:

[0067] Calculate the loss of the labeled packet : Instance probability function Parameterization is performed, using Platt scaling to convert the anomaly scores obtained from the autoencoder into probabilistic representations, and the model accuracy is jointly calibrated by associating them with bag labels. The calibrated probabilities after transformation are:

[0068] Calibration parameters The converted instance scores are within the range [0,1]. When converting instance probabilities to packet probabilities, instances are sorted according to their anomaly scores and assigned corresponding weights. The probability of a packet being anomalous is determined by the different weight values ​​of the instances and the highest and lowest anomalous probabilities among them. The final anomalousness level of the packet is:

[0069] Instance feature learning is used for label correction to avoid the model overfitting to the positive class. The ratio of positive class bags to the most reliable negative bags is approximately 1:3 to balance the model's classification performance and improve the reliability of the results.

[0070]

[0071] Here, λ is a weighting parameter used to balance the two types of losses. The network parameters, including the encoder, decoder, and probability mapping parameters, are optimized using the backpropagation algorithm to effectively detect potential data anomalies.

[0072] Step 4.4: In each round, continuously select the most reliable negative and positive packets and input them into the supervised model for training to obtain the optimal parameters and output the comprehensive anomaly label for all instances.

[0073] Step 5: Process the tabular data to be detected using the model trained in Step 4; the model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data according to a preset threshold, and finally outputs the anomaly detection results, including but not limited to a list of abnormal data, anomaly type labeling, and anomaly ratio statistics.

[0074] Implementation Method 2: A Tabular Data Anomaly Detection System Based on Genetic Algorithm Learning Paradigm The system includes an encoding weighting module, a screening module, a crossover and mutation module, a training module, and a prediction module. The encoding weighting module obtains the dirty dataset to be detected, combines the word vector model and the TF-IDF algorithm to obtain the initial encoding form of the data, and uses the synergistic effect of the context attention mechanism to complete the encoding weighting to obtain the encoded representation of the data instance; The screening module is used to construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection. The crossover and mutation module performs crossover and mutation operations on the individuals in the population divided by the screening module, and introduces a population diversity fusion strategy to screen individuals, and finally completes the labeling of high-probability normal individuals and abnormal individuals. The training module is based on the sample set labeled by the cross-mutation module. It dynamically constructs positive and negative class sample bags through an improved label diffusion mechanism, expands the training data during the adaptive iterative generation of pseudo labels, and performs model training. The prediction module processes the tabular data to be detected using the model trained by the training module; the model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data according to a preset threshold, and finally outputs the anomaly detection result.

[0075] An electronic device includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the above method.

[0076] A computer-readable storage medium for storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0077] The memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the methods described in this invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0078] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means such as coaxial cable, optical fiber, digital subscriber line, DSL, or wireless means such as infrared, wireless, microwave, etc. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium such as a floppy disk, hard disk, magnetic tape; an optical medium such as a high-density digital video disc, DVD; or a semiconductor medium such as a solid-state disk, SSD, etc.

[0079] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0080] It should be noted that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied as execution by a hardware decoding processor, or as execution by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above methods.

[0081] The foregoing has provided a detailed description of the tabular data anomaly detection method and system based on the genetic algorithm learning paradigm proposed in this invention, and has elucidated the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for detecting anomalies in tabular data based on a genetic algorithm learning paradigm, characterized in that: The method specifically includes the following steps: Step 1: Obtain the dirty dataset to be detected, combine the word vector model and TF-IDF algorithm to obtain the initial encoding form of the data, and use the synergistic effect of the context attention mechanism to complete the encoding weighting to obtain the encoded representation of the data instance; Step 2: Construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection; Step 3: Perform crossover and mutation operations on the individuals in the population divided in Step 2, and introduce a population diversity fusion strategy to screen individuals, and finally complete the labeling of high-probability normal individuals and abnormal individuals. Step 4: Based on the sample set labeled in Step 3, dynamically construct positive and negative class sample bags by improving the label diffusion mechanism, expand the training data during the adaptive iterative generation of pseudo labels, and execute model training. Step 5: Process the tabular data to be detected using the model trained in Step 4; the model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data according to a preset threshold, and finally outputs the anomaly detection result.

2. The detection method according to claim 1, characterized in that: Step 1 is as follows: Step 1.1: Obtain the dirty dataset of the table. For each cell instance, use Word2vec to calculate the word vector and use the TF-IDF algorithm to obtain the term frequency-inverse file encoding. Step 1.2: Concatenate the word vectors and word frequency codes, and then weight them according to the context attention mechanism to obtain the final encoded vector.

3. The detection method according to claim 2, characterized in that: Step 2 is as follows: Step 2.1: Decompose the dataset into sets of tuples by row, and treat each tuple as an independent individual in the genetic algorithm; Step 2.2: Randomly select a subset of k tuples from the set of tuples, and use a comprehensive fitness function to evaluate the degree of abnormality of each cell; The comprehensive fitness function integrates three key anomaly detection components, including a function dependency violation anomaly score based on the TANE algorithm, a statistical outlier detection score, and an adaptive mode anomaly detection score. Step 2.3: Use the TANE algorithm-based function dependency violation detection method to obtain attribute partitions and value partitions, and label the instance labels in the qualitative data according to the final generated rules; Step 2.4: Use statistical-based quantitative data anomaly detection, combining Gaussian distribution and histogram, to generate outlier labels for the quantitative data portion; Step 2.5: Use a self-learning pattern detection method to dynamically generate a set of regular expression rules, establish a column-level dominant pattern representation, and divide it into dominant pattern, secondary pattern and noise pattern, thereby labeling abnormal tags that do not conform to the expected format. Step 2.6: Based on the fitness in Step 2.5, complete the population initialization and individual selection.

4. The detection method according to claim 3, characterized in that: Step 3 specifically involves: Step 3.1: Perform a crossover operation on the randomly selected tuple individuals in Step 2.

2. Exchange some contents of the paired individuals according to their fitness differences, prioritizing the exchange of cells with larger fitness differences, and generate new tuples. Randomly select tuples with high fitness for mutation operation, and randomly select new values ​​from candidate cells in the same column to replace the mutated tuples. Step 3.2: By analyzing the distribution characteristics of cells in each column within the population, the selection weights are dynamically adjusted to perform diversity fusion selection of the population. The average fitness of all cells is calculated as the baseline threshold to obtain high-fitness and low-fitness populations. Individuals with a high probability of being normal or abnormal are labeled.

5. The detection method according to claim 3, characterized in that: Step 4 is as follows: Step 4.1: Based on the labeled sample set in Step 3.2, the sample features and label labels are obtained. The cosine similarity metric is used to calculate the distance between unlabeled samples and labeled samples in the column level. For unlabeled samples, they are selected and assigned to the set of labels of the labeled samples with the highest cosine similarity. Step 4.2: Divide the set into groups. Each positive package consists of a fixed proportion of positive instances in the column and a random number of global positive instances. Each negative package consists of a fixed proportion of negative instances in the column and a random number of global negative instances. Generate a negative package-level instance set. Step 4.3: Reconstruct the negative bag set using an autoencoder, minimize the reconstruction error, and calculate the loss function to obtain the loss value on the negative bag set; Select the set of negative packets with the smallest reconstruction error as the most reliable negative packets, and input them along with the positive packets as the dataset into the supervised network for training. Output the anomalous probability of each instance in each package and map it to the anomalous probability value of the package, and use backpropagation to minimize the loss; Step 4.4: In each round, continuously select the most reliable negative and positive packets and input them into the supervised model for training to obtain the optimal parameters. Output the comprehensive anomaly label for all instances to complete the model training.

6. A tabular data anomaly detection system based on a genetic algorithm learning paradigm, characterized in that: The system is used to execute the tabular data anomaly detection method based on the genetic algorithm learning paradigm as described in any one of claims 1 to 5; The system includes an encoding weighting module, a screening module, a crossover and mutation module, a training module, and a prediction module. The encoding weighting module obtains the dirty dataset to be detected, combines the word vector model and the TF-IDF algorithm to obtain the initial encoding form of the data, and uses the synergistic effect of the context attention mechanism to complete the encoding weighting to obtain the encoded representation of the data instance; The screening module is used to construct a data selector based on a genetic algorithm, decompose the dataset into tuple individuals, calculate the fitness score of each individual through a designed comprehensive fitness function, and complete the population initialization and individual selection. The crossover and mutation module performs crossover and mutation operations on the individuals in the population divided by the screening module, and introduces a population diversity fusion strategy to screen individuals, and finally completes the labeling of high-probability normal individuals and abnormal individuals. The training module is based on the sample set labeled by the cross-mutation module. It dynamically constructs positive and negative class sample bags through an improved label diffusion mechanism, expands the training data during the adaptive iterative generation of pseudo labels, and performs model training. The prediction module processes the table data to be detected using the model trained by the training module. The model outputs the anomaly probability or anomaly label for each data tuple, filters out abnormal data based on a preset threshold, and finally outputs the anomaly detection result.

7. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium for storing computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images