Real-time anomaly detection and intelligent early warning system and method based on artificial intelligence

Abstract

The application discloses a real-time anomaly detection and intelligent early warning system and method based on artificial intelligence, relates to the technical field of artificial intelligence and data security, and comprises the following steps: acquiring historical data assets and constructing a tensor potential function; guiding the migration of generated detector particles through the tensor gradient of the tensor potential function; filtering the migrated detector particles through negative selection to generate an initial detector, performing cloning on the initial detector to generate a cloned detector, and migrating to a potential abnormal area based on the tensor gradient; when the cloned detector stably resides in the potential abnormal area, recording the evolution track of the cloned detector and constructing a trend memory unit; generating a trend detector according to the trend memory unit and migrating to an area where the detector particles are insufficient based on the tensor gradient; and performing response matching on the to-be-detected data assets, the cloned detector and the trend detector to complete anomaly detection and early warning. The application ensures high detection accuracy and early warning response rate in a complex data environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and data security, in particular to a real-time anomaly detection and intelligent early warning system and method based on artificial intelligence. BACKGROUND

[0002] Under the background of the rapid development of current data-driven business, a large amount of structured and unstructured data assets have been accumulated in various information systems, covering multiple dimensions such as field content, scheduling records, and label identification. These data assets support intelligent analysis, prediction modeling, and decision execution, and have become the core resources supporting organizational operation and business optimization. However, with the continuous growth of data volume and the increasing complexity of processing chain, data anomaly problems have become increasingly prominent, often manifested as data structure disorder, processing path anomaly, and sovereignty compliance risk. These anomalies, if not identified and addressed in a timely manner, can easily lead to system performance degradation, business logic errors, and even information security incidents.

[0003] Existing anomaly detection methods generally rely on static rules or fixed threshold models, which are difficult to adapt to the dynamic changes of heterogeneous data structures, multi-source data blood relationship, and sovereignty attributes, and have significant problems such as weak generalization ability, response lag, and insufficient coverage. In addition, most traditional methods use point-to-point detection mechanisms and lack modeling of data processing evolution trends, resulting in low identification accuracy for potential risk areas and difficulty in achieving real-time, accurate, and scalable anomaly identification. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the technical problem solved by the present application is that the existing multi-source heterogeneous data has weak potential anomaly area identification ability, insufficient detector coverage, and lagging early warning response.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a real-time anomaly detection and intelligent early warning method based on artificial intelligence, comprising:

[0007] Obtain historical data assets and construct a tensor potential function;

[0008] Guide the migration of generated detector particles through the tensor gradient of the tensor potential function;

[0009] Filter the migrated detector particles through negative selection, generate an initial detector, execute cloning on the initial detector to generate a cloned detector, and migrate to the potential anomaly area based on the tensor gradient;

[0010] When the clone detector stably resides in the potentially anomalous region, the evolution trajectory of the clone detector is recorded and a trend memory unit is constructed; based on the trend memory unit, a trend detector is generated and migrated to the region with insufficient detector particle coverage based on the tensor gradient;

[0011] The data assets to be detected are matched with the responses of the clone detector and the trend detector to complete the anomaly detection and early warning.

[0012] As a preferred embodiment of the real-time anomaly detection and intelligent early warning method based on artificial intelligence described in this invention, the historical data assets include structured and unstructured fields, task scheduling records, and data tag information.

[0013] Feature vectors are extracted from structured and unstructured fields and standardized into historical feature vectors; historical data processing paths are reconstructed based on task scheduling records to construct data lineage chains; sensitive attributes and compliance identifiers of historical data assets are parsed from data tag information and mapped to generate sovereign information tensors.

[0014] Using the recording order of historical data assets as an index, multiple sample points are generated sequentially to form a sample point set; each sample point is formed by concatenating historical feature vectors, data lineage chains, and sovereignty information tensors; each sample point is compressed into a vector of fixed dimension d using MLP, and the feature space R is obtained based on the vector. d The location of the sample points in the data.

[0015] As a preferred embodiment of the real-time anomaly detection and intelligent early warning method based on artificial intelligence described in this invention, the tensor potential function is constructed as follows: the local density anomaly is calculated using the local outlier method through the historical feature vector of each sample point; the structural entropy is calculated as the path complexity through the data lineage chain of each sample point; and the sovereign risk intensity is generated by weighted fusion of the sovereign information tensor attributes of each sample point.

[0016] The local density anomaly degree, path complexity, and sovereign risk intensity of each sample point are combined according to a preset weighting factor to form a single-point anomaly potential value. The distance between any position in the feature space and the sample point is calculated using a Gaussian kernel function to construct a tensor potential energy function. The potential energy function values ​​of all sample points are sorted in order to form a set of potential energy function values. The sovereign information tensor attributes include data sensitivity, supervision strictness, and cross-border flow level.

[0017] As a preferred embodiment of the real-time anomaly detection and intelligent early warning method based on artificial intelligence described in this invention, the generation of the detector particles includes: when the potential energy function value is greater than the dynamic quantile threshold θ min Uniform sampling of N in the region initA feature space position point is taken as the initial position of a detector particle, and the detector particle is generated; wherein θ min is the qth percentile of the potential function set of all sample points; N init represents the number of detector particles generated.

[0018] The detector particle migration includes: calculating the gradient of the tensor potential function according to the current position of the detector particle, multiplying the gradient by a damping coefficient, adding the current particle position, and superimposing a disturbance term subject to a Gaussian distribution with a mean of 0 and a variance of σ 2 , to obtain an updated position; when the difference between the potential function values of any detector particle in two adjacent iterations is < the average fluctuation amplitude of the potential function value, it is determined to be converged, and the iteration is stopped

[0019] As a preferred scheme of the real-time anomaly detection and intelligent early warning method based on artificial intelligence, the initial detector is generated by screening normal sample points from the sample point set to form a normal sample point set, calculating the mean center point and feature space standard deviation of the normal sample point set, and setting the normal region determination radius to k times the feature space standard deviation; the Euclidean distance between each detector particle and the mean center point is calculated, and when the Euclidean distance > the normal region determination radius, the detector particle is retained as an initial detector; otherwise, the detector particle is removed.

[0020] The cloned detector is generated by generating N c cloned detectors for each initial detector, the position of each cloned detector being the position of the initial detector plus a cloned Gaussian disturbance term subject to a mean of 0 and a variance of The gradient of the tensor potential function is calculated according to the current position of the cloned detector, and migration is performed according to the cloned Gaussian disturbance term; when the position of the cloned detector corresponds to a tensor potential function value ≥ the gth percentile of the potential function set, it is determined that the cloned detector is in a potential abnormal region; wherein N c represents the number of cloned detectors generated.

[0021] As a preferred scheme of the real-time anomaly detection and intelligent early warning method based on artificial intelligence, when the cloned detector is in the potential abnormal region for L consecutive iterations, it is determined that the cloned detector is stably resident; the position iteration trajectory of the cloned detector is recorded, and the trajectory length is not more than T max , wherein T max represents the upper limit of the trajectory length; the position iteration trajectories of all stably resident cloned detectors are collected to construct the trend memory unit.

[0022] As a preferred scheme of the real-time anomaly detection and intelligent early warning method based on artificial intelligence, wherein: the trend detector comprises: extracting a trend hotspot area according to the position iterative trajectory of the stable resident clone detector, randomly sampling an initial position of the trend detector in the trend hotspot area, and performing migration update according to the gradient of the tensor potential function to obtain a final trend detector position; and constructing a space division graph based on the position, sorting the area of each space division graph in ascending order, and defining the areas ranked in the top p% of the area as insufficient coverage areas; if the distance between the trend detector position and the clone detector in any insufficient coverage area is less than a preset spatial radius threshold, it is determined that the trend detector falls into the insufficient coverage area; the extraction of the trend hotspot area comprises: extracting the trajectory points of the position iterative trajectory of the stable resident clone detector and performing density clustering to cluster the center points to construct a spherical area with a radius of v times the cluster variance; and the union of all spherical areas is taken as the trend hotspot area.

[0023] The anomaly detection and early warning comprises: calculating the response of the data asset to be detected and each clone detector and each trend detector by using the inverse of the Euclidean distance, accumulating all response calculation values as the total response strength, and determining that an anomaly exists and triggering intelligent early warning when the total response strength exceeds a response threshold; the intelligent early warning comprises recording an abnormal event, pushing alarm information, and triggering data asset control.

[0024] As a preferred scheme of the real-time anomaly detection and intelligent early warning system based on artificial intelligence, wherein: the tensor potential function module acquires historical data assets and constructs a tensor potential function;

[0025] The detector module guides the migration of the generated detector particles through the tensor gradient of the tensor potential function, filters the migrated detector particles through negative selection, generates an initial detector, clones the initial detector to generate a clone detector, and migrates the clone detector to a potential abnormal area based on the tensor gradient;

[0026] The trend memory module records the evolution trajectory of the clone detector when the clone detector stably resides in the potential abnormal area, and constructs a trend memory unit;

[0027] The coverage module generates a trend detector based on the trend memory unit and migrates the trend detector to a detector particle insufficient coverage area based on the tensor gradient;

[0028] The anomaly detection module performs response matching of the data asset to be detected and the clone detector and the trend detector to complete anomaly detection and early warning.

[0029] A computer device comprises a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement steps of a real-time anomaly detection and intelligent early warning method based on artificial intelligence.

[0030] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement steps of a real-time anomaly detection and intelligent early warning method based on artificial intelligence.

[0031] The beneficial effects of the present application: the real-time anomaly detection and intelligent early warning method based on artificial intelligence provided by the present application realizes accurate characterization of potential abnormal areas of historical data by constructing a tensor potential function that integrates local density anomaly degree, data blood structure complexity and sovereign risk intensity; the detector particles are guided to migrate by the tensor gradient, so that the detector can adaptively focus on the high abnormal potential area, and the coverage range and spatial flexibility of anomaly detection are enhanced by the negative selection strategy and dynamic cloning mechanism; after the cloned detector is stably resident, its evolution trajectory is recorded, a trend memory unit is constructed to extract abnormal hotspots, further supplement the particle coverage blind area, and ensure high detection accuracy and early warning response rate in complex data environment. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0033] Figure 1 The overall flowchart of the real-time anomaly detection and intelligent early warning method based on artificial intelligence provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0035] Embodiment 1, refer to Figure 1 For an embodiment of the present application, a real-time anomaly detection and intelligent early warning method based on artificial intelligence is provided, comprising:

[0036] S1: Acquire historical data assets and construct a tensor potential function; guide the migration of generated detector particles through the tensor gradient of the tensor potential function.

[0037] Historical data assets include: structured data fields: numerical and categorical data fields; unstructured data fields: text, image, audio and video fields, etc.; task scheduling records: task scheduling logs of data processing flow (including task ID, execution time, and task type); data tag information: fields such as sensitivity level, regulatory compliance requirements, and cross-border circulation level.

[0038] Structured and unstructured fields are preprocessed separately (e.g., text embedding, image feature extraction) and then unified into a fixed-dimensional historical feature vector f. i Based on the task ID and execution timestamp recorded in the task scheduling log, the task execution order is restored to generate a clear data lineage structure L. line,i ; Clearly defined sensitive attributes, regulatory stringency, and cross-border circulation levels are extracted from data tag information and unified into a three-dimensional numerical sovereignty information tensor L. son,i .

[0039] Based on the recording order of historical data assets, and taking each record as a sample point, a corresponding single sample structure is generated:

[0040] x i =[f i ,L line,i ,L son,i ]

[0041] The above splicing structure x i The input is uniformly fed into a pre-trained multilayer perceptron (MLP), which maps it to a vector of uniform dimension.

[0042] x i ′ =MLP(x i ),x i ′ ∈R d

[0043] The position vector X′ of each sample point in the feature space = {x i ′}.

[0044] Where, x i f represents the concatenated feature vector of the i-th sample point; i L represents the historical feature vector of the i-th sample point; line,i L represents the data lineage vector of the i-th sample point; son,i The vector represents the sovereign information tensor of the i-th sample point; MLP(·) represents the Multi-Layer Perceptron model; xi ′ represents the embedding vector of the i-th sample point after MLP mapping; d represents the dimension of the unified embedding vector; X' = {x i '} represents the position vector set of all sample points in the feature space.

[0045] Each sample point x i ', f i , L line,i , L son,i Processing process: calculate three potential energy respectively: local density anomaly degree S(f i ): use local outlier factor LOF to calculate the local density anomaly degree of each sample point: S(f i ) = LOF(f i ).

[0046] Data bloodline chain structure entropy Ey(L line,i ): take the data bloodline chain as the node type sequence, calculate the Shannon entropy of the type to represent the path complexity:

[0047]

[0048] Sovereign risk intensity Rk(L son,i )): adopt a weighted way to fuse sensitivity, regulatory strictness, cross-border level:

[0049] Rk(L son,i ) = w1·Sy + w2·RS + w3·CBL

[0050] Combine the above three indicators to form the single-point anomaly potential value of each sample point:

[0051] Q i = α1S(f i ) + α2Ey(L line,i ) + α3Rk(L son,i )

[0052] Wherein, S(f i ) represents the local density anomaly degree of the i-th sample point LOF(·) represents the local outlier factor (Local Outlier Factor) calculation function; data bloodline chain structure entropy Ey(L line,i ) represents the data bloodline chain structure entropy of the i-th sample point represents the full set of all bloodline node types p(c) represents the probability of the occurrence of a type node c in the bloodline chain logp(c) represents the logarithmic value of the type probability (with natural logarithm as the base); c represents the index; Rk(L son,i) represents the main risk intensity of the i-th sample point, w1, w2, w3 represent the weight coefficients of sensitivity, regulatory strictness, and cross-border level respectively, Sy represents the data sensitivity score (Sensitivity), RS represents the regulatory strictness score (Regulation Strictness), and CBL represents the cross-border level score (Cross-Border Level); Q i represents the comprehensive abnormal potential value of the i-th sample point; α1, α2, α3 represent the fusion weight coefficients of the three indicators, respectively, for adjusting the proportion of their influence in the potential value.

[0053] Diffusing single-point abnormal potential values to any point in the feature space using a Gaussian kernel Tensor potential function:

[0054]

[0055] wherein the Gaussian kernel function is defined as:

[0056]

[0057] wherein Φ(x) represents the potential energy value of the potential energy function at position x in the feature space; x represents any to-be-evaluated point in the feature space; xi represents the i-th sample point; and σ represents the kernel width of the Gaussian kernel. wherein the potential energy value of the i-th sample point is obtained by the following formula: i represents the Gaussian kernel similarity between position x and the i-th sample point.

[0058] Ordering the potential energy function values of all sample points in sequence to form a potential energy function value set.

[0059] Generating the detector particles includes uniformly sampling N min feature space position points as initial positions of the detector particles in the region where the potential energy function value is greater than the dynamic quantile threshold θ init ; and generating the detector particles at the points; wherein θ min is the q% quantile of the potential energy function set of all sample points; N init represents the number of detector particles generated.

[0060] Gradient based on the tensor potential function Calculating the update path of the particle, and the formula representing the specific update rule is:

[0061]

[0062] represents the position vector of the j-th detector particle in the feature space at the t+1 iteration; β represents the damping coefficient; represents the gradient of the tensor potential function Φ(x) at position ; ε trepresents a random disturbance term added in the tth iteration; represents a Gaussian distribution with mean 0 and variance σ 2 represents a Gaussian distribution with mean 0 and variance σ 2 represents the variance of the Gaussian disturbance;

[0063] When the difference between the potential energy function values of any detector particle in two adjacent iterations is less than or equal to the average fluctuation amplitude of the potential energy value, the migration of the detector particle is completed.

[0064] In the processing of structured and unstructured data assets, by respectively extracting features and vectorizing embedding of numerical, categorical fields and unstructured fields such as text and images, and combining the three-dimensional sovereign information tensor constructed by the data processing path structure and sensitive attribute field restored from the task scheduling log, a unified dimensional feature splicing sample is formed. Compared with the traditional method of using only structured fields or part of the metadata to construct samples, this method can comprehensively integrate static data features, dynamic task behavior chains and sovereign supervision dimensions, thereby improving the completeness and discriminability of sample expression.

[0065] In the modeling of abnormal potential, the local density anomaly degree is calculated by introducing the local outlier factor (LOF), the complexity of the task path is evaluated using the path structure entropy, and the sensitivity, supervision strictness and cross-border level are fused in a weighted manner to generate a single-point abnormal potential value for each sample point. Then, the single-point potential is diffused to the entire feature space using a Gaussian kernel to construct a continuous and derivable tensor potential function. Compared with the method of discrete judgment based on fixed threshold or category label, this scheme can accurately describe the data distribution trend and local disturbance risk in high-dimensional continuous space, thereby improving the accuracy and interpretability of potential abnormal area modeling.

[0066] In the generation and migration of detectors, the gradient information of the tensor potential function is calculated to guide the force field migration of the generated detector particles in the high potential area, and the stable convergence of the particle position is realized under the dynamic disturbance and damping balance mechanism. This scheme not only avoids the generalization problem caused by random distribution of initial detectors, but also adaptively adjusts the initial position based on the spatial distribution law of historical samples, so that the subsequent detection coverage is more concentrated in the potential abnormal area, thereby effectively improving the initial detection ability and subsequent response efficiency.

[0067] S2: filtering the migrated detector particles by negative selection, generating an initial detector, cloning the initial detector to generate a cloned detector, and migrating to the potential abnormal area based on the tensor gradient.

[0068] In the historical data asset, the record samples meeting the normal behavior characteristics are filtered to construct a normal sample set, and the characteristics include: the data label information is identified as compliant; the data blood relationship path structure is stable, and there is no abnormal processing record in history; the sovereign information tensor is characterized by low sensitivity, high regulatory intensity and no cross-border circulation.

[0069] The sample points meeting the above conditions are defined as normal sample points to form a normal sample point set

[0070] The feature space mean center point μ and standard deviation vector σ of the normal sample point set are calculated ; the normal region judgment radius r=k·||σ|| is set, where k is a preset multiple coefficient; for each migrated detector particle z j , the Euclidean distance / / z j -μ|| between it and the center point μ is calculated; when the distance / / z j -μ||>r, the particle is retained as an initial detector; otherwise, the particle is rejected; after the above filtering, an initial detector set is formed

[0071] k represents the set normal region radius multiple coefficient; r=k· / / σ / / represents the normal region radius for judgment; z j represents the position vector of the jth detector particle; / / z j -μ / / represents the Euclidean distance between the detector particle and the center point; represents the initial detector set retained by negative selection filtering.

[0072] For each initial detector m cloned detectors are generated 2. The initial position of each cloned detector is generated by superimposing a cloned disturbance term on the original detector position:

[0073] where

[0074] Perform tensor gradient guided cloning migration on each cloned detector Perform the following update iteration:

[0075]

[0076] where β represents the damping coefficient of the gradient direction; represents the gradient of the tensor potential function; ∈ (j,t) is the cloned Gaussian disturbance superimposed in each round.

[0077] where z irepresents the position vector of the i-th initial detector; m represents the number of detectors to be cloned for each initial detector; represents the initial position of the j-th cloned detector (0th iteration); ∈ (j) represents the cloned migration position perturbation term; represents that the perturbation term obeys a Gaussian distribution with mean 0 and variance ; represents the variance of the cloned perturbation term (cloned perturbation strength); represents the position of the j-th cloned detector of the i-th initial detector at the t-th iteration; represents its position at the next iteration; β represents the damping coefficient of the gradient of the tensor potential function; represents the gradient of the tensor potential function; ∈ (j,t) represents the cloned perturbation term at the t-th iteration; T clone represents the maximum number of iteration steps of cloned migration; ∈ conv represents the convergence judgment threshold of the cloned migration process; represents the amplitude of the potential energy change between two consecutive iterations; represents the final stable position of the cloned detector (converged position).

[0078] After each step of iteration, the potential energy change between two consecutive iterations is calculated:

[0079]

[0080] If the convergence criterion is met or the step limit is reached, it is determined that the cloned detector completes migration;

[0081] The converged position is recorded as

[0082] The potential energy values of all sample points form a set After sorting, the g% quantile potential energy value Φ thresh is obtained.

[0083] If , it is determined that the cloned detector is in a potential abnormal region and is retained, otherwise it is removed.

[0084] wherein, represents the set of potential energy values Φ thresh of all sample points in the feature space; represents the g% quantile of the set (i.e. dynamic threshold); represents the potential function value of the final position of the cloned detector; g represents the set abnormal region judgment quantile threshold.

[0085] In the initial detector screening aspect, by constructing the "normal sample point set" under the structured condition, the three types of information of data label, blood path and principal right tensor are jointly used to accurately extract the data records of normal historical behavior in multiple source dimensions. Further based on the set, the center point and standard deviation vector are calculated in the feature space, and by setting the Euclidean distance limit (defined as the judgment radius in the standard deviation multiple), the negative selection filtering of the detector particles is realized. Compared with the existing method based on rule hard coding or random migration strategy, this method can effectively constrain the spatial position distribution of the initial detector, so that the subsequent detection is no longer focused on the obviously normal area, reducing the false positive rate and resource waste.

[0086] In the cloning detector generation and migration phase, by generating multiple cloned detectors with Gaussian perturbation for each initial detector, the problem of limited coverage ability of single-point detector particles is overcome; at the same time, the tensor potential function gradient is introduced in the cloning process, and the random perturbation is superimposed to form the migration trajectory, and the stable residence position of the cloned detector is determined by the convergence criterion (such as the potential energy change amplitude less than the set threshold). Compared with the traditional static model or fixed window moving strategy, this method can dynamically adapt to the complex distribution situation in the feature space, improve the exploration efficiency and flexibility of the detection area.

[0087] In the potential abnormal area judgment link, the tensor potential function value corresponding to the convergence position is compared with the preset dynamic quantile threshold to ensure that the cloned detector finally focuses on the relatively high-risk area. This method upgrades the static threshold judgment method to a mechanism based on dynamic adjustment of data distribution, and combines the continuous convergence behavior to control the risk of mismigration, significantly enhancing the effectiveness of the detector and the discriminability of the convergence area, providing a high-quality initial basis for subsequent trend tracking and response matching.

[0088] S3: When the cloned detector stably resides in the potential abnormal area, record the evolution trajectory of the cloned detector and construct the trend memory unit; generate a trend detector according to the trend memory unit and migrate to the area covered by the detector particles based on the tensor gradient.

[0089] When the cloned detector is continuously iterated for L times in the potential abnormal area, it is considered that the cloned detector stably resides; record the position iteration trajectory of the cloned detector, and the trajectory length is not more than T max , wherein T max represents the upper limit of the trajectory length; all the trajectories of the stably residing cloned detectors are uniformly collected to construct the trend memory unit.

[0090] Extract the evolutionary trajectory of all converged clone detectors from the trend memory unit, collect all trajectory points in each trajectory during the migration process to form a trajectory point set. Perform a density-based clustering algorithm (such as DBSCAN) on the trajectory point set, extract the high-density clustering area of all trajectory points, and calculate the center point position and variance of each cluster.

[0091] Center the center point of each cluster as the center, and the v times of the cluster variance as the radius to build a spherical region. The union of all spherical regions is defined as the trend hotspot region, which reflects the migration focus direction of historical abnormal behavior.

[0092] Random sampling in the trend hotspot region generates the initial position of several trend detectors. For each trend detector, a gradient-driven mechanism of tensor potential function is used to perform a limited number of migration processes to update the position vector of the trend detector. The migration update process is as follows:

[0093]

[0094] Where x t represents the position vector of the trend detector at the tth iteration; x t+1 represents the position vector of the trend detector after the t+1th iteration; represents the gradient of the tensor potential function at x t ; γ represents the damping coefficient in the migration process, which controls the gradient guide intensity; t represents the Gaussian disturbance term of the tth iteration; represents t obeys the normal distribution with mean 0 and variance ; represents the disturbance intensity in the tth migration.

[0095] The migration process of each trend detector is completed within a set maximum number of steps T trend or terminated when the change amplitude of the tensor potential function value is lower than the average convergence fluctuation threshold.

[0096] Finally, judge whether the convergence position of the trend detector is located in the feature space region not covered by the current detector particle set. If the condition is met, the trend detector is included in the trend detector set for subsequent supplementary abnormal detection coverage.

[0097] Record the convergence position of all existing stable resident clone detectors as set D det , and construct a Voronoi diagram based on the set. Count the area of each Voronoi region, and arrange all region area values in ascending order. Take the top p% area as the "undercover area", and its corresponding detection

[0098] radius threshold of whether the trend detector enters the under-coverage area; x' trend represents the position vector of the trend detector after convergence; p% represents the area sorting percentage of the under-coverage area selection.

[0099] Most of the existing anomaly detection techniques rely on static features at the current time or one-time sampling method to construct a detector set, lack the use of historical detector behavior path, and are difficult to identify the evolution trend of abnormal activities. The above process is based on the stable residence behavior of the cloned detector, and when L is located in the potential abnormal area, the complete evolution trajectory is recorded, and a trend memory unit is constructed. By aggregating multiple trajectory points and performing density clustering analysis, the system can automatically extract the evolution focusing direction of multiple historical anomaly detection paths, form a trend hotspot area, reflect the migration of historical abnormal behavior in the feature space, and effectively enhance the modeling and understanding ability of the system to non-static abnormal form.

[0100] The existing cloning strategy usually generates new detectors by global Gaussian disturbance, but lacks directional guidance, which is easy to cause detector drift invalid or coverage overlap. Uniform sampling is performed in the trend hotspot area to generate the initial position of the trend detector, and the gradient driving mechanism of the tensor potential function is used for directional force migration. Gaussian disturbance is introduced in the migration process to maintain the exploratory, so that the detector can be efficiently migrated to the potential low-coverage area in the feature space under the guidance of the historical evolution path of the abnormal trend. This method can significantly reduce the probability of new detectors falling into high-density areas, and improve the coverage uniformity and response sensitivity of the detector to the abnormal trend space.

[0101] Traditional detector distribution often lacks coverage rationality evaluation means, resulting in local area redundancy and key area omission. The mechanism introduces Voronoi diagram analysis method, constructs the spatial distribution graph of all converged detector positions, and calculates the perception area of each detector according to the area. The position set of the under-coverage area is extracted by ascending order sorting. After the migration of the trend detector is completed, whether it fills the vacancy is judged according to the Euclidean distance between its convergence position and the under-coverage area. If it meets the requirements, it is included in the detector set. In this way, structural detector layout optimization under the guidance of trend can be realized, ensuring that the coverage of the detector in the space is more reasonable in breadth and depth, and improving the robustness and coverage stability of the overall anomaly detection.

[0102] S4: Perform response matching on the to-be-detected data assets with the cloned detector and the trend detector to complete anomaly detection and early warning.

[0103] Receive the current to-be-detected data assets, wherein each record is processed by the same feature processing method as the historical data assets to generate a uniform dimension to-be-detected sample point set:

[0104]

[0105] The system loads the current set of detectors, including:

[0106] Cloning detector collection: Each detector position is

[0107] Trend detector set: Each detector position is

[0108] For each sample point to be tested The system calculates its response value in relation to all clone detectors and trend detectors:

[0109] (1) Cloning detector response:

[0110]

[0111] (2) Trend detector response:

[0112]

[0113] The total response intensity is obtained by summing the response values ​​of each sample point.

[0114]

[0115] Set response threshold θ r ,when When, determine X test Abnormal data triggers an intelligent early warning mechanism. The response threshold θ r The value can be set based on the statistical quantiles of the total response intensity distribution of the normal sample point set. For example, the p% quantile of the normal response distribution can be taken as θ. r To achieve dynamic and sensitive judgment.

[0116] Among them, X test This represents a sample set consisting of the data assets to be tested. Let represent the i-th sample point to be detected, with dimension d, belonging to R. d N represents the total number of sample points to be tested; This represents the set of clone detectors, containing n items. c One clone detector; Let represent the position vector of the i-th clone detector in the feature space, belonging to R. d ; This represents a set of trend detectors, containing n items. t A trend detector; denotes the position vector of the k-th trend detector in the feature space, belonging to R d ; denotes the response value between the i-th sample point and the j-th clone detector; denotes the square of the Euclidean distance between the sample point and the clone detector; denotes the variance parameter of the clone detector response function, controlling the sensitivity of the response function to distance; exp(·) denotes the exponential function, used to map the distance to a similarity value in [0, 1] (the smaller the distance, the higher the response); denotes the response value between the i-th sample point and the k-th trend detector; denotes the square of the Euclidean distance between the sample point and the trend detector; denotes the variance parameter of the trend detector response function; denotes the total response intensity of the i-th sample point on all detectors (clone + trend); denotes the sum of the response values with all clone detectors; denotes the sum of the response values with all trend detectors; θ r denotes the pre-warning threshold of the total response intensity; if the i-th sample is considered as an abnormal point, triggering the pre-warning mechanism; p% denotes the statistical quantile setting, for example, set as the 95th quantile of the response intensity distribution of normal samples.

[0117] When the intelligent pre-warning is triggered, the system automatically performs the following linkage operations:

[0118] 1. Record the current abnormal data asset identification, abnormal timestamp, and corresponding clone detector response record.

[0119] 2. Push the alarm information to the visual monitoring interface or pre-warning subscription end.

[0120] 3. Linkage trigger data asset control mechanism, including suspension of downstream processing flow, reporting to compliance system, or performing isolation operation, etc.

[0121] The detector set composed of clone detectors and trend detectors is used for response matching of the data assets to be detected, which can effectively cover different types of potential abnormal patterns. Among them, the clone detector is derived from the local replication and migration of the initial detector, with high spatial distribution density, suitable for finding centralized anomalies; the trend detector is derived from the trend memory feedback mechanism, with trend directionality and regional repair ability, suitable for covering boundary areas and response blank areas. By fusing the two detector sets in the response function and uniformly calculating the response intensity, the detection coverage and response sensitivity can be significantly improved, thereby improving the accuracy of the overall anomaly detection.

[0122] The Euclidean distance is mapped to a standardized response value by using a Gaussian function calculation mode for the detector response value, and the total response intensity of each test sample is obtained by weighted summation of all detector response values. At the same time, the statistical quantile of the response intensity distribution of the normal sample point set is used to dynamically set the response threshold, so as to avoid the problem of false judgment of the fixed threshold under different data distributions. In the case of periodicity, heterogeneity or data drift of the data asset feature structure, the dynamic determination mode can automatically adjust the determination boundary according to the historical normal distribution, so as to ensure the stability and applicability of the abnormal detection.

[0123] When the total response intensity exceeds the determination threshold, the system automatically triggers a multi-level intelligent early warning response mechanism, including recording the abnormal detector and data identification information, pushing the visual alarm event, and triggering the asset control operation. The response value driven mechanism can make the alarm event have specific detector correlation, so as to improve the subsequent fault positioning capability and response explanation. At the same time, the early warning linkage mechanism supports automatic processing flow such as data isolation, permission freezing or early warning notification, reduces the manual response cost and improves the risk control efficiency, which is beneficial to realize the rapid closed loop of abnormal detection and response in the actual production system.

[0124] Embodiment 2, which is an embodiment of the present application, provides a real-time abnormal detection and intelligent early warning method based on artificial intelligence. In order to verify the beneficial effects of the present application, economic benefit calculation and simulation experiment are used for scientific demonstration.

[0125] In this experiment, the historical data assets of a city data governance platform in the fourth quarter of 2023 are taken as the experimental objects, and the data includes user access log, ETL scheduling record, API calling track, field compliance label and other multiple dimensions, a total of 1000 historical data records are collected. The experiment first extracts 10-dimensional standardized features from the structured fields to construct a unified feature vector set; then the scheduling path reconstruction tool is used to restore the data blood relationship chain, and the path complexity is measured by information entropy; at the same time, the data label is parsed to generate a sovereignty information tensor including "data sensitivity", "supervision intensity" and "cross-border level". The above features are spliced to form the complete input vector of each sample point, and are compressed into a fixed dimension representation in the feature space through the MLP model.

[0126] Based on the above sample point set, the system constructs a tensor potential function, which fuses the local outlier factor, path entropy and sovereign risk intensity to calculate the single-point potential value. The experiment uses Gaussian kernel function to estimate the potential field distribution, and uniformly samples the particles in the region with potential higher than 90% quantile. All particles migrate in the direction according to the potential gradient, and the convergence condition is that the continuous potential change is lower than the average fluctuation amplitude. After the particles converge, the negative selection algorithm is applied to remove the particles falling into the normal region, and only the high deviation detectors are retained to construct the initial detector set.

[0127] Subsequently, each initial detector derives multiple cloned detectors and guides them to migrate to the region with high potential function value. The system sets continuous stable residence conditions, records the residence trajectory, and performs density clustering to construct the trend hotspot area, forming a trend detector. The system finally loads the current cloned detector and trend detector set to calculate the abnormal response of the new data batch. The response adopts an inverse mapping function of the Euclidean distance, and the response value exceeding the threshold set based on the 95% quantile of the normal sample triggers an early warning, and the system automatically records the anomaly, reports the alarm, and suspends the downstream call of the data.

[0128] Number of historical samples: 1000; sample dimension: 10-dimensional structured + 1-dimensional path entropy + 3-dimensional sovereign tensor; number of initial detectors: 80; total number of cloned detectors: 800 (10 each); number of trend detectors: 45; new batch of samples to be detected: 300.

[0129] The response results are as follows:

[0130] Number of data entries determined to be abnormal: 42, accounting for 14.00%.

[0131] Abnormalities independently identified by trend detectors: 11, accounting for 26.19%.

[0132] Samples hit by multiple detectors: 21, accounting for 50.00%.

[0133] Response value threshold (95% normal quantile): 0.1467.

[0134] Maximum response intensity: 0.2784, minimum: 0.0243, mean: 0.1082.

[0135] From the experimental data, it can be observed that the method has significant advantages in terms of abnormal detection accuracy and response efficiency. First, in the sample point generation link, through the triple fusion modeling of structured features, path entropy and sovereign tensor, compared with traditional methods based only on feature similarity or isolation forest, this method can more accurately capture and comply with the implicit high-risk features related to sensitivity. In this experiment, 12 abnormal data were identified by this method due to the high sensitivity of sovereign attributes, but there was no obvious abnormality in the conventional features.

[0136] The detector particle migration strategy guided by the tensor potential function realizes the rapid focusing of the detector in the high-risk field, and can avoid the redundancy and misjudgment caused by traditional grid sampling or manual setting of positions. Through the combination mechanism of potential energy gradient migration + Gaussian disturbance, the average iteration of 80 initial detectors converging only needs 8 steps, which is much lower than the baseline method (average 13 steps), effectively improving the model initialization efficiency.

[0137] Furthermore, the introduction of the trend memory mechanism and the trend detector significantly enhances the system's ability to identify "structural evolution anomalies". For example, a certain field may have moderate sensitivity in the early stage but frequent aggregation in the recent period. Traditional methods often ignore this trend change due to insufficient window. However, the invention identifies this trend change through trajectory evolution clustering, successfully captures multiple trend risk points, and improves the overall detection coverage by about 18%.

[0138] Finally, in the response matching and early warning mechanism, a dynamic response threshold is used instead of a fixed standard. The lower limit of the response is dynamically adjusted in combination with the current detector coverage capability, effectively controlling the false positive rate (the false positive rate in this experiment is 3.2%, which is much lower than the 7.9% of a certain commercial model). The linkage control mechanism also realizes the automatic response after early warning, greatly improving the practicality of the system.

[0139] In summary, this experiment fully verifies that the method performs well in terms of reliability, sensitivity, and adaptability in the task of anomaly detection and intelligent early warning in a high-dimensional data environment, and has wide engineering feasibility and practical application value.

[0140] Embodiment 3, which is an embodiment of the present application, provides an artificial intelligence-based real-time anomaly detection and intelligent early warning system, comprising:

[0141] The detector module guides the migration of the generated detector particles through the tensor gradient of the tensor potential function; filters the migrated detector particles through negative selection to generate an initial detector; performs cloning on the initial detector to generate a cloned detector, and migrates to the potential abnormal area based on the tensor gradient.

[0142] The trend memory module records the evolution trajectory of the cloned detector when the cloned detector stably resides in the potential abnormal area, and constructs a trend memory unit.

[0143] The coverage module generates a trend detector based on the trend memory unit and migrates to the area where the detector particles are insufficient based on the tensor gradient.

[0144] The anomaly detection module performs response matching between the data assets to be detected and the cloned detector and the trend detector to complete anomaly detection and early warning.

[0145] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0146] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0147] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0148] It should be understood that portions of the present application can be implemented with hardware, software, firmware, or combinations thereof. In the embodiments described above, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be understood that the foregoing embodiments are merely illustrative of the present application and that modifications or equivalents can occur to one skilled in the art without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that all the implementations described above and modifications and / or additions made thereto without departing from the spirit and scope of the present application can be included in the scope of the present application.

[0149] It should be understood that the foregoing embodiments are merely illustrative of the present application and that modifications or equivalents can occur to one skilled in the art without departing from the spirit and scope of the present application as defined by the appended claims. It should be understood that all the implementations described above and modifications and / or additions made thereto without departing from the spirit and scope of the present application can be included in the scope of the present application.

Claims

1. A real-time anomaly detection and intelligent early warning method based on artificial intelligence, characterized in that, include: Acquire historical data assets and construct a tensor potential function; The construction process of the tensor potential function is as follows: the local density anomaly is calculated using the local outlier method through the historical feature vector of each sample point; the structural entropy is calculated as the path complexity through the data lineage chain of each sample point; and the sovereign risk intensity is generated by weighted fusion of the sovereign information tensor attributes of each sample point. The local density anomaly degree, path complexity, and sovereign risk intensity of each sample point are combined according to a preset weighting factor to form a single-point anomaly potential value. The distance between any position in the feature space and the sample point is calculated using a Gaussian kernel function to construct a tensor potential energy function. The potential energy function values ​​of all sample points are sorted in order to form a set of potential energy function values. The sovereign information tensor attributes include data sensitivity, supervision strictness, and cross-border flow level. The tensor gradient of the tensor potential function guides the migration of the generated detector particles. The negative selection filters the migrated detector particles to generate an initial detector. The initial detector is cloned to generate a cloned detector, which is then migrated to the potential anomalous region based on the tensor gradient. When the clone detector stably resides in a potentially anomalous region, the evolutionary trajectory of the clone detector is recorded, and a trend memory unit is constructed. Based on the trend memory unit, a trend detector is generated and migrated to the region where the detector particles are under-covered based on the tensor gradient. The data assets to be detected are matched with the responses of the clone detector and the trend detector to complete the anomaly detection and early warning. The anomaly detection and early warning includes calculating the response of the data asset to be detected to each clone detector and each trend detector using the inverse ratio of Euclidean distance, and summing all the calculated response values ​​as the total response intensity. When the total response intensity exceeds the response threshold, it is determined to be abnormal and an intelligent early warning is triggered; the intelligent early warning includes recording abnormal events, pushing alarm information, and triggering data asset control in conjunction with the event.

2. The real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in claim 1, characterized in that: The historical data assets include structured and unstructured fields, task scheduling records, and data tag information; Feature vectors are extracted from structured and unstructured fields and standardized into historical feature vectors; historical data processing paths are reconstructed based on task scheduling records to construct data lineage chains; sensitive attributes and compliance identifiers of historical data assets are parsed from data tag information and mapped to generate sovereign information tensors. Using the recording order of historical data assets as an index, multiple sample points are generated sequentially to form a sample point set; each sample point is formed by concatenating historical feature vectors, data lineage chains, and sovereignty information tensors; each sample point is compressed into a vector of fixed dimension d using MLP, and the feature space R is obtained based on the vector. d The location of the sample points in the data.

3. The real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in claim 2, characterized in that: Generating the detector particles includes, when the potential energy function value is greater than the dynamic quantile threshold θ min Uniform sampling of N in the region init θ is used as the initial position of the detector particle, and the detector particle is generated; where θ min N is the q% quantile of the potential energy function set for all sample points; init Indicates the number of particles generated by the detector; The detector particle migration includes calculating the gradient of the tensor potential function based on the current position of the detector particle, multiplying the gradient by a damping coefficient, adding it to the current particle position, and then superimposing the result with a mean of 0 and a variance of σ. 2 The perturbation term of the Gaussian distribution is used to obtain the updated position; when the difference in potential function value of any detector particle in two adjacent iterations is less than the average fluctuation amplitude of the potential function value, it is determined to be converged and the iteration is stopped.

4. The real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in claim 3, characterized in that: The generation of the initial detector includes selecting normal sample points from the sample point set to form a normal sample point set; Calculate the mean center point and feature space standard deviation of the normal sample point set, and set the normal region judgment radius to be k times the feature space standard deviation; Calculate the Euclidean distance between each detector particle and the mean center point. When the Euclidean distance is greater than the normal region determination radius, retain the detector particle as the initial detector. Otherwise, remove the detector particles; The generated clone detector includes generating N from each initial detector. c There are 1 clone detector, and the position of each clone detector is determined by adding a sum to the position of the initial detector, which follows a mean of 0 and a variance of . The cloned Gaussian perturbation term is used; the gradient of the tensor potential function is calculated based on the current position of the clone detector, and migration is performed based on the cloned Gaussian perturbation term; when the tensor potential function value corresponding to the position of the clone detector is ≥ the g% quantile of the potential function set, the clone detector is considered to be in a potential anomaly region; where N c This indicates the number of clone detectors generated.

5. The real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in claim 4, characterized in that: The clone detector is considered to be stably residing when it is in the potential anomalous region for L consecutive iterations. Record the position iteration trajectory of the clone detector, the trajectory length of which does not exceed T. max , among which, T max Indicates the upper limit of the trajectory length; The trend memory unit is constructed by summarizing the position iteration trajectories of all stably residing clone detectors.

6. The real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in claim 5, characterized in that: The generation of the trend detector includes: extracting trend hotspot regions based on the position iteration trajectory of the stably resident clone detector; randomly sampling and generating the initial position of the trend detector within the trend hotspot region; and performing migration update based on the gradient of the tensor potential function to obtain the final trend detector position; and constructing a spatial partition map based on the position, sorting the area of ​​each spatial partition map in ascending order, and defining the area ranking in the top p% as the under-coverage area. If the distance between the trend detector location and any clone detector within an insufficient coverage area is less than a preset spatial radius threshold, then the trend detector is determined to have fallen into an insufficient coverage area. The extraction of trend hotspot regions includes extracting the trajectory points of the position iteration trajectory of the stably resident clone detector and performing density clustering to construct a spherical region with a radius equal to a multiple of the cluster variance v, centered on the cluster center point; and combining all spherical regions as the trend hotspot region.

7. A system employing the real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in any one of claims 1 to 6, characterized in that: The tensor potential energy function module acquires historical data assets and constructs tensor potential energy functions. The detector module guides the migration of the generated detector particles through the tensor gradient of the tensor potential energy function. Negative selection filters the migrated detector particles to generate the initial detector; The initial detector is cloned to generate a clone detector, which is then migrated to the potential anomaly region based on the tensor gradient. The trend memory module records the evolution trajectory of the clone detector and constructs trend memory units when the clone detector stably resides in a potentially anomalous region. The coverage module generates a trend detector based on the trend memory unit and migrates to the region where the detector particles are not covered based on the tensor gradient. The anomaly detection module matches the data assets to be detected with the responses of the clone detector and trend detector to complete the anomaly detection and early warning.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the real-time anomaly detection and intelligent early warning method based on artificial intelligence as described in any one of claims 1 to 6.

Images