An abnormal data monitoring method and system based on artificial intelligence

Abstract

The present invention discloses an abnormal data monitoring method and system based on artificial intelligence, which relates to the field of data monitoring technology. The method generates a standardized feature matrix by real-time monitoring and preprocessing data from different sources, effectively reduces data noise and scale differences between features, and analyzes the deviation of each sample relative to the overall distribution, and combines the critical value of the chi-square distribution table to make abnormal judgments on the samples, thereby ensuring the accuracy of abnormal data detection. Then, artificial intelligence technology is used to enhance the abnormal samples, and by introducing distribution constraints, several groups of supplementary abnormal samples are dynamically generated, while ensuring that the generated samples are close to the real distribution, and then by identifying and screening outliers in the abnormal group, the unreasonable samples introduced by noise or deviation in the enhancement process are further reduced, thereby improving the quality of the abnormal group data.

Description

Technical Field

[0001] The present invention relates to the technical field of data monitoring, and in particular to an abnormal data monitoring method and system based on artificial intelligence. Background Art

[0002] Artificial intelligence (AI), as an important research direction in the field of information science, has received extensive attention and development in application fields such as data analysis, anomaly detection, and intelligent monitoring in recent years. In the specific application of artificial intelligence, the abnormal data monitoring method is a key technology, which is mainly used in scenarios such as the Internet of Things, industrial manufacturing, financial risk control, and medical health. In these fields, the monitoring of abnormal data is indispensable. Specifically, with the development of Internet of Things devices and big data technology, the amount of multi-source heterogeneous data generated by different monitoring devices has exploded. How to identify anomalies in these complex data in real time and efficiently has become an important research direction in data analysis.

[0003] In the existing abnormal data monitoring technology, although the algorithms based on statistical methods and machine learning have been widely used, they still face many challenges. First, abnormal data often accounts for a low proportion of the overall data set, and usually has the characteristics of sparsity and dynamic distribution changes, which makes the existing methods vulnerable to data imbalance problems, making the detection model prefer normal data much more than abnormal data, which makes it easy to miss key abnormal behaviors, normal data is incorrectly marked as abnormal, and the model's generalization ability for unknown abnormal behaviors is insufficient. These consequences directly affect the stability and security of the system, and its negative impact is particularly serious. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides an abnormal data monitoring method and system based on artificial intelligence, which solves the problems in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an abnormal data monitoring method based on artificial intelligence, comprising the following steps:

[0006] S1. Use monitoring equipment to monitor data from different sources in real time to generate the original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ;

[0007] S2, based on the standardized feature matrix obtained in S1 , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within

[0008] S3: Using artificial intelligence technology, combined with the identification content in S2, perform sample enhancement on abnormal samples, and introduce distribution constraints during sample enhancement to obtain several groups of supplementary abnormal samples. , and combined with the abnormal samples identified in S2, an abnormal group is generated;

[0009] S4. Based on the abnormal group, analyze several groups of supplementary abnormal samples The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd;

[0010] S5. Preset a balance threshold K and compare and analyze it with the balance degree Phd to measure the current balance state of the entire sample distribution, and based on the measurement result, determine whether it is necessary to perform iterative enhancement operations.

[0011] Preferably, the specific steps of S1 include:

[0012] S11. Monitor and record data from different sources to generate the original feature matrix , where, from different sources means from different monitoring devices;

[0013] S12, the original feature matrix in S11 Perform data preprocessing, where data preprocessing includes the original feature matrix All feature parameters in the data are normalized and standardized, and the feature parameters after data preprocessing are summarized to construct a standardized feature matrix , where the standardized feature matrix Includes each characteristic parameter in each sample.

[0014] Preferably, the specific steps of S2 include:

[0015] S21, based on the standardized feature matrix obtained in S12 , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz, which is obtained by the following formula:

[0016] ;

[0017] In the formula, represents the distribution deviation factor of the i-th sample; m represents the number of characteristic parameters in the corresponding sample, j=1, 2, 3, ..., m, represents the value of the jth characteristic parameter in the i-th sample, Represents the standardized feature matrix The mean value of the jth characteristic parameter in Represents the standardized feature matrix The standard deviation of the jth feature parameter, i represents the standardized feature matrix The number of samples within .

[0018] Preferably, the specific step S2 also includes:

[0019] S22, using the chi-square distribution table, pre-set the critical value P, and compare it with the distribution deviation factor Fpyz to identify the standardized feature matrix The specific identification contents of abnormal samples are as follows:

[0020] S221, if the distribution deviation factor Fpyz exceeds the critical value P, then the current standardized feature matrix is ​​judged The corresponding samples are in an abnormal state and are marked as abnormal samples;

[0021] S222: If the distribution deviation factor Fpyz does not exceed the critical value P, the current normalized feature matrix is ​​determined. The corresponding samples are not in abnormal state and are marked as normal samples.

[0022] Preferably, the specific steps of S3 include:

[0023] S31, according to the method of identifying abnormal samples in S22, several groups of abnormal samples are counted, and the groups of abnormal samples are sorted according to the numerical value of the distribution deviation factor Fpyz, and the abnormal sample with the largest distribution deviation factor Fpyz is extracted as the enhanced reference sample By performing sample enhancement operations on several groups of abnormal samples and introducing distribution constraints during the sample enhancement process, we can obtain additional abnormal samples. , the supplementary abnormal samples The specific formula is as follows:

[0024] ;

[0025] In the formula, represents the enhanced benchmark sample; Represents the enhanced benchmark sample A sample randomly selected from the neighborhood of Represents the distribution center of all abnormal samples; Represents a random number; represents the regularization coefficient; Represents the enhanced benchmark sample To the distribution center of all abnormal samples The square distance indicates the degree to which the sample deviates from the center of the distribution; represents the offset in the neighborhood direction; k and g both represent a sample in several groups of abnormal samples;

[0026] S32, by repeated random sampling and To generate several groups of supplementary abnormal samples ;

[0027] S33, by respectively obtaining a plurality of groups of abnormal samples and a plurality of groups of supplementary abnormal samples in S31 and S32 Statistics are performed to generate an abnormal group, and statistics are performed on the normal samples marked in S222 to generate a normal group.

[0028] Preferably, the specific steps of S4 include:

[0029] S41. Analyze several groups of supplementary abnormal samples based on the abnormal group The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The out-of-group coefficient Lqyz is obtained as follows:

[0030] ;

[0031] In the formula, represents the outlier coefficient of the a-th supplementary abnormal sample, represents the bth abnormal sample in the neighbor set of the ath supplementary abnormal sample, represents the neighbor set of the a-th supplementary abnormal sample, represents the ath supplementary abnormal sample, represents the local reachable density of the a-th supplementary abnormal sample, represents the number of abnormal samples in the neighbor set of the a-th supplementary abnormal sample, a represents the number of the supplementary abnormal sample, and b represents the number of the abnormal sample in the neighbor set of the a-th supplementary abnormal sample; express The local reachable density reflects The density level within the neighbor set.

[0032] Preferably, the specific step S4 also includes:

[0033] S42. Based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter outliers in the abnormal group. The specific contents are as follows:

[0034] S421, if the out-of-group coefficient Lqyz of the corresponding supplementary abnormal sample is ≥ 1, then determine the corresponding supplementary abnormal sample is an outlier, and the additional abnormal sample Remove from the outlier group;

[0035] S422: If the out-of-group coefficient Lqyz of the corresponding supplementary abnormal sample is less than 1, then the corresponding supplementary abnormal sample is determined. is not an outlier, and the additional abnormal sample considered as abnormal samples.

[0036] Preferably, the specific step S4 also includes:

[0037] S43. On the basis of screening out outliers, re-count the abnormal groups and analyze the balance of the entire sample distribution after the sample enhancement operation to obtain the balance degree Phd. The balance degree Phd is obtained by the following formula:

[0038] ;

[0039] In the formula, Zcs represents the number of samples in the normal group, and Ycs represents the number of samples in the abnormal group.

[0040] Preferably, the specific steps of S5 include:

[0041] S51, pre-set a balance threshold K, and compare and analyze the balance degree Phd with the balance threshold K to measure the current balance state of the entire sample distribution. The specific measurement content is as follows:

[0042] S511. If the balance degree Phd exceeds the balance threshold K, the entire sample distribution is considered to be not in a balanced state and a red warning is issued;

[0043] S512, if the balance degree Phd does not exceed the balance threshold K, the entire sample distribution is measured to be in a balanced state and a green warning is issued;

[0044] S52. Based on the measurement results in S51, when a green warning is received, there is no need to perform iterative enhancement operations, and the abnormal group will be input into the cloud platform; when a red warning is received, steps S3 and S4 will be repeated to perform iterative enhancement operations until a green warning is received, at which time the iterative enhancement operations are stopped and the abnormal group is input into the cloud platform.

[0045] An abnormal data monitoring system based on artificial intelligence, including a data acquisition module, an identification module, a supplement module, an analysis and processing module and an intelligent feedback module;

[0046] The data acquisition module is used to monitor data from different sources in real time using monitoring equipment to generate an original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ;

[0047] The recognition module is used to obtain a standardized feature matrix , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within

[0048] The supplementary module is used to use artificial intelligence technology, combined with the identification content in the identification module, to perform sample enhancement operations on abnormal samples, and introduce distribution constraints during the sample enhancement process to obtain several groups of supplementary abnormal samples , and combined with the identified abnormal samples, generate an abnormal group;

[0049] The analysis and processing module analyzes several groups of supplementary abnormal samples based on the abnormal group. The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd;

[0050] The intelligent feedback module is used to pre-set a balance threshold K and compare and analyze it with the balance degree Phd to measure the current balance state of the entire sample distribution, and based on the measurement results, determine whether it is necessary to perform iterative enhancement operations.

[0051] The present invention provides an abnormal data monitoring method and system based on artificial intelligence, which has the following beneficial effects:

[0052] (1) Through steps S1 and S2, this method monitors and preprocesses data from different sources in real time, generates a standardized feature matrix, effectively reduces data noise and scale differences between features, and improves data quality. In S2, the deviation of each sample relative to the overall distribution is accurately analyzed by calculating the distribution deviation factor Fpyz, and the sample is judged as abnormal in combination with the critical value of the chi-square distribution table, ensuring the accuracy of abnormal data detection. In step S3, artificial intelligence technology is used to enhance abnormal samples. By introducing distribution constraints, several groups of supplementary abnormal samples are dynamically generated, while ensuring that the generated samples are close to the real distribution. This method effectively expands the potential distribution range of abnormal samples by controlling the position of supplementary samples between abnormal benchmark samples and neighborhoods and the degree of deviation from the distribution center. At the same time, it enhances the sensitivity of the model to different types of abnormal data, and improves the generalization ability and robustness of the model. In step S4, the outlier coefficient Lqyz is calculated to identify and filter out outliers in the abnormal group, further reducing the unreasonable samples introduced by noise or deviation during the enhancement process, thereby improving the quality of the abnormal group data. By calculating the balance degree Phd, the ratio distribution of abnormal samples and normal samples is comprehensively evaluated, and the sample distribution structure is further optimized. In step S5, the balance degree is dynamically evaluated through the preset balance threshold and it is determined whether it is necessary to iterate the enhancement operation, which ensures the balance of the final data distribution, effectively solves the problem of data imbalance, and provides a high-quality sample set for subsequent model training.

[0053] (2) In step S2, by calculating the distribution deviation factor Fpyz, the degree of deviation of each sample relative to the overall sample distribution is analyzed to accurately reflect the distance of each sample to the distribution center. Combined with the critical value set in the chi-square distribution table, the abnormal state of each sample is judged to ensure the scientificity and rigor of the detection. This method combines the chi-square distribution in statistics and effectively solves the problems of high false alarm rate and false negative rate in existing methods. It is especially suitable for high-dimensional data and diversified feature distribution scenarios.

[0054] (3) Generate supplementary abnormal samples through step S3, and introduce distribution constraints in the sample enhancement process to ensure that the generation of supplementary abnormal samples is scientific and reasonable. This enhancement method based on distribution constraints effectively solves the distribution deviation and sample distortion problems caused by simple interpolation in traditional enhancement technology. Through step S33, the enhanced supplementary abnormal samples are merged with the original abnormal samples using statistical methods to generate abnormal groups, and normal samples are statistically analyzed to generate normal groups: the enhancement of the abnormal group compensates for the sparsity of abnormal data and optimizes the data structure; the statistics of the normal group ensures data integrity. This grouping strategy provides more balanced input data for the anomaly detection model in the data processing stage, solves the problems of scarcity of abnormal samples and data imbalance, and further improves the detection performance and stability of the model.

[0055] (4) By calculating the outlier coefficient of the supplementary abnormal samples, the degree of outlier of each supplementary abnormal sample relative to the neighborhood distribution is effectively evaluated. This density-based evaluation method can dynamically identify possible noise or invalid samples in the generated samples, thereby improving the overall credibility of the supplementary abnormal samples. At the same time, outliers that may exist in the abnormal group are identified and eliminated. This method effectively solves the problems of abnormal distribution deviation and data noise that may be introduced in the sample enhancement process, optimizes the data quality of the abnormal group, and ensures the rationality and distribution consistency of the enhanced samples. By analyzing and screening the outliers of the supplementary abnormal samples, this method avoids the distribution deviation or data imbalance caused by low-quality samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of an abnormal data monitoring method based on artificial intelligence of the present invention;

[0057] Figure 2 This is a block diagram of an abnormal data monitoring system based on artificial intelligence in the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Example 1

[0060] See also Figure 1 The present invention provides an abnormal data monitoring method based on artificial intelligence, comprising the following steps:

[0061] S1. Use monitoring equipment to monitor data from different sources in real time to generate the original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ;

[0062] S2, based on the standardized feature matrix obtained in S1 , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within

[0063] S3: Using artificial intelligence technology, combined with the identification content in S2, perform sample enhancement on abnormal samples, and introduce distribution constraints during sample enhancement to obtain several groups of supplementary abnormal samples. , and combined with the abnormal samples identified in S2, an abnormal group is generated;

[0064] S4. Based on the abnormal group, analyze several groups of supplementary abnormal samples The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd;

[0065] S5. Preset a balance threshold K and compare and analyze it with the balance degree Phd to measure the current balance state of the entire sample distribution, and based on the measurement result, determine whether it is necessary to perform iterative enhancement operations.

[0066] In this embodiment, the method first collects and preprocesses the data, and in step S2, based on the standardized feature matrix Calculating the distribution deviation factor Fpyz and setting the critical value in combination with the chi-square distribution table can further accurately identify abnormal samples. This method effectively improves the detection sensitivity and accuracy of abnormal samples by comprehensively analyzing the degree of deviation of each sample relative to the entire sample distribution. In step S3, the abnormal samples are enhanced by artificial intelligence technology, and distribution constraints are introduced to control sample generation, so that the supplementary abnormal samples are as close to the actual data distribution as possible, avoiding the generated samples from deviating from the distribution center. This process generates samples through dynamic interpolation and regularization constraints, expands the distribution range of abnormal data, improves the generalization ability of the detection model for different types of anomalies, and effectively alleviates the problem of data imbalance. In step S4, the degree of outliers of the supplementary abnormal samples is quantitatively analyzed by using the outlier coefficient Lqyz, and outliers in the abnormal group are eliminated in combination with the local reachable density, further optimizing the quality of the abnormal group. By screening outliers with excessive deviations, noise interference in the enhanced samples is avoided, thereby improving the representativeness and reliability of the enhanced data. In step S5, by setting the balance threshold and comparing the balance degree, the balance state of the sample distribution is evaluated in real time to ensure the rationality of the final data set. Once the sample distribution is detected to be unbalanced, the system will automatically trigger the iterative enhancement job to further optimize the sample distribution structure. This dynamic enhancement strategy not only improves the robustness of the detection model, but also effectively avoids the misjudgment problem caused by distribution deviation in traditional static enhancement methods. In summary, this method monitors and optimizes abnormal data from multiple dimensions through artificial intelligence technology, which not only improves the accuracy of abnormal sample detection, but also ensures the diversity and balance of data through enhancement and distribution optimization, thereby improving the overall performance of the monitoring system.

[0067] Example 2

[0068] Please refer to Figure 1 , specifically: S1 specific steps include:

[0069] S11. Monitor and record data from different sources to generate the original feature matrix , where, from different sources means from different monitoring devices;

[0070] S12, the original feature matrix in S11 Perform data preprocessing, where data preprocessing includes the original feature matrix All feature parameters in the data are normalized and standardized, and the feature parameters after data preprocessing are summarized to construct a standardized feature matrix , where the standardized feature matrix Includes each characteristic parameter in each sample.

[0071] The specific steps of S2 include:

[0072] S21, based on the standardized feature matrix obtained in S12 , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz, which is obtained by the following formula:

[0073] ;

[0074] In the formula, represents the distribution deviation factor of the i-th sample, reflecting the distance from the i-th sample to the center of the entire sample distribution; m represents the number of characteristic parameters in the corresponding sample, j=1, 2, 3, ..., m, represents the value of the jth characteristic parameter in the i-th sample, Represents the standardized feature matrix The mean value of the jth characteristic parameter in Represents the standardized feature matrix The standard deviation of the jth feature parameter, i represents the standardized feature matrix The number of samples within .

[0075] Normalized feature matrix Each sample can be understood as a number of sets of characteristic parameters monitored each time;

[0076] The specific steps of S2 also include:

[0077] S22, using the chi-square distribution table, pre-set the critical value P, and compare it with the distribution deviation factor Fpyz to identify the standardized feature matrix The specific identification contents of abnormal samples are as follows:

[0078] S221, if the distribution deviation factor Fpyz exceeds the critical value P, then the current standardized feature matrix is ​​judged The corresponding samples are in an abnormal state and are marked as abnormal samples;

[0079] S222: If the distribution deviation factor Fpyz does not exceed the critical value P, the current normalized feature matrix is ​​determined. The corresponding samples are not in abnormal state and are marked as normal samples.

[0080] Among them, the chi-square distribution table is a statistical table used to find the cumulative distribution function value of the chi-square distribution (Chi-Square Distribution). It is usually used to determine the threshold under the significance level (such as the threshold determination used in anomaly detection); the chi-square distribution is a right-skewed distribution (asymmetric) and is widely used in hypothesis testing and variance analysis in statistics.

[0081] In this embodiment, in step S1, the data from different monitoring devices are monitored and recorded to generate an original feature matrix , and normalize and standardize the feature parameters in the original feature matrix to construct a standardized feature matrix , this data preprocessing process can effectively eliminate the dimensional differences and scale effects between different feature parameters, reduce data noise, improve the quality and consistency of data, and lay a solid foundation for subsequent analysis and anomaly detection. At the same time, the standardized feature matrix enables the eigenvalues ​​of each sample to be calculated at the same scale, which improves the accuracy of deviation analysis and anomaly detection. In step S2, by calculating the distribution deviation factor Fpyz, the degree of deviation of each sample relative to the center of the entire sample distribution is quantitatively reflected. Based on the calculation of the standardized feature matrix, the mean and standard deviation of each feature parameter are fully considered, and the abnormality of the sample is comprehensively evaluated, avoiding the misjudgment problem caused by relying only on a single feature or simple rules in traditional methods. At the same time, by introducing the chi-square distribution table to set the critical value and perform comparative analysis, the samples can be scientifically and reasonably divided into abnormal samples and normal samples, ensuring that the detection results have higher reliability and interpretability. Using the chi-square distribution table to set the critical value for comparative analysis simplifies the screening process of abnormal samples, and the abnormal state can be quickly identified by simple numerical comparison. Especially in large-scale data scenarios, this method can effectively reduce the computational complexity and improve the efficiency of anomaly detection. Through the above process, this method not only improves the accuracy of data processing and analysis, but also enhances the scientificity, reliability and efficiency of anomaly detection, providing a solid foundation for subsequent data enhancement and anomaly management.

[0082] Example 3

[0083] Please refer to Figure 1 , specifically: S3 specific steps include:

[0084] S31, according to the method of identifying abnormal samples in S22, several groups of abnormal samples are counted, and the groups of abnormal samples are sorted according to the numerical value of the distribution deviation factor Fpyz, and the abnormal sample with the largest distribution deviation factor Fpyz is extracted as the enhanced reference sample By performing sample enhancement operations on several groups of abnormal samples and introducing distribution constraints during the sample enhancement process, we can obtain additional abnormal samples. , the supplementary abnormal samples The specific formula is as follows:

[0085] ;

[0086] In the formula, represents an enhanced reference sample to serve as a reference point for generating supplementary abnormal samples; Represents the enhanced benchmark sample A sample randomly selected from the neighborhood of Represents the distribution center of all abnormal samples (i.e., the mean vector of abnormal samples); Represents a random number used to dynamically control the interpolation ratio; Represents the regularization coefficient, which is used to adjust the influence strength of the regularization term; Represents the enhanced benchmark sample To the distribution center of all abnormal samples The square distance indicates the degree to which the sample deviates from the center of the distribution; Indicates the offset in the neighborhood direction and supplements abnormal samples Indicates that the generated enhanced sample is an abnormal sample Based on , new samples are generated by interpolation method (combining the distribution center of neighborhood samples and all abnormal samples). The goal is to generate new samples that are as close to the real data distribution as possible, while covering the potential distribution range of more abnormal samples. k and g both represent a sample in several groups of abnormal samples; Controlling for abnormal samples in and neighborhood The position between. is a regularization term that controls the degree to which the generated samples deviate from the center of the distribution.

[0087] S32, by repeated random sampling and To generate several groups of supplementary abnormal samples ;

[0088] S33, by respectively obtaining a plurality of groups of abnormal samples and a plurality of groups of supplementary abnormal samples in S31 and S32 Statistics are performed to generate an abnormal group, and statistics are performed on the normal samples marked in S222 to generate a normal group.

[0089] Among them, the abnormal sample with the largest distribution deviation factor Fpyz is extracted as the enhanced benchmark sample , is to enhance samples with high abnormality in a targeted manner to improve the model's sensitivity to boundary samples.

[0090] Impact: Control supplement abnormal samples exist and The position between Impact of: Controlling for supplementary abnormal samples Biased toward the center of distribution of all abnormal samples By introducing distribution constraints during sample enhancement, Needs to be greater than 0, indicating that regularization has been added to supplement abnormal samples Will move to the distribution center of all abnormal samples offset to achieve distribution constraints;

[0091] In this embodiment, by sorting the distribution deviation factor Fpyz of the abnormal samples, extracting the abnormal samples with the largest deviation as the enhanced reference samples, it is possible to effectively locate the representative boundary samples for enhancement. This targeted enhancement method further avoids excessive repeated sampling of ordinary abnormal samples, improves the sensitivity of the model to the boundary abnormal samples, and thus enhances the model's ability to detect anomalies, especially for sparsely distributed or complexly distributed abnormal samples. In the sample enhancement process, distribution constraints are introduced to dynamically generate supplementary abnormal samples. This enhancement method based on distribution constraints ensures that the supplementary samples can not only expand the potential distribution range of abnormal samples, but also further avoid excessive deviation of the generated samples from the actual distribution. By repeatedly sampling randomly to generate several groups of supplementary abnormal samples, the diversity of abnormal group samples is further enhanced. While optimizing the distribution of abnormal data, this process effectively alleviates the problem of unbalanced data distribution: the richness of abnormal group samples: the generation of supplementary samples enhances the diversity of abnormal data and expands the adaptability of the model in complex abnormal scenarios. Balance between abnormal and normal groups: The enhanced samples and normal samples are grouped and counted to construct a reasonable sample distribution, providing a more balanced data structure for subsequent model training. By selecting the sample with the maximum value of the distribution deviation factor as the reference point and introducing distribution constraints, this method effectively solves the problems of oversampling and noise introduction in traditional sample enhancement methods. At the same time, the supplementary samples cover more potential abnormal patterns and reduce the false alarm rate; the enhanced distribution constraints ensure the rationality of the distribution of the supplementary samples and reduce the false alarm rate.

[0092] Example 4

[0093] Please refer to Figure 1 , specifically: S4 specific steps include:

[0094] S41. Analyze several groups of supplementary abnormal samples based on the abnormal group The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The out-of-group coefficient Lqyz is obtained as follows:

[0095] ;

[0096] In the formula, represents the outlier coefficient of the a-th supplementary abnormal sample, represents the bth abnormal sample in the neighbor set of the ath supplementary abnormal sample, represents the neighbor set of the a-th supplementary abnormal sample, represents the ath supplementary abnormal sample, represents the local reachable density of the a-th supplementary abnormal sample, represents the number of abnormal samples in the neighbor set of the a-th supplementary abnormal sample, a represents the number of the supplementary abnormal sample, and b represents the number of the abnormal sample in the neighbor set of the a-th supplementary abnormal sample; express The local reachable density reflects The density level within the neighbor set;

[0097] The neighbor set of the a-th supplementary abnormal sample mentioned above It refers to a cluster closest to the a-th supplementary abnormal sample, which can be obtained according to the clustering algorithm; there are multiple groups of abnormal samples in the neighbor set. This is one of the abnormal samples;

[0098] S411, wherein the local reachable density of the a-th supplementary abnormal sample Obtained by the following formula:

[0099] ;

[0100] In the formula, represents the number of abnormal samples in the neighbor set of the a-th supplementary abnormal sample, express The bth abnormal sample in The maximum spacing of abnormal samples except the b-th abnormal sample in, express and The spacing between

[0101] S412, among which, The local reachable density Obtained by the following formula:

[0102] ;

[0103] In the formula, express The mean of all abnormal samples in .

[0104] The specific steps of S4 also include:

[0105] S42. Based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter outliers in the abnormal group. The specific contents are as follows:

[0106] S421: If the outlier coefficient Lqyz of the corresponding supplementary abnormal sample does not fall within the preset threshold, the corresponding supplementary abnormal sample is judged to be is an outlier, and the additional abnormal sample Remove from the outlier group;

[0107] S422: If the outlier coefficient Lqyz of the corresponding supplementary abnormal sample falls within the preset threshold, the corresponding supplementary abnormal sample is judged to be is not an outlier, and the additional abnormal sample considered as abnormal samples.

[0108] Among them, the preset threshold range is near the value 1. Specifically, when the outlier coefficient Lqyz of the corresponding supplementary abnormal sample is greater than or less than 1, it means that the local density of the supplementary abnormal sample deviates from the average density of its neighborhood, indicating that the supplementary abnormal sample is more likely to be an outlier because its density in the data distribution is much lower than that of the surrounding data points; when the outlier coefficient Lqyz of the corresponding supplementary abnormal sample is close to 1, it means that the local density of the supplementary abnormal sample is close to the local density of its neighborhood, which indicates that the supplementary abnormal sample is not an outlier, but a data point that is "normal and consistent" with other sample points in the neighborhood.

[0109] In this embodiment, the degree of outlier of each supplementary abnormal sample in the abnormal group can be quantitatively analyzed by calculating the outlier coefficient Lqyz in step S41. The combined calculation method takes into account the density difference between the supplementary abnormal sample and its neighbor set. This process further reduces the impact of noise and meaningless samples that may be introduced in the enhancement process, ensures the rationality and representativeness of the distribution of the supplementary abnormal samples, and thus improves the overall quality of the abnormal group. The supplementary abnormal samples are screened out of outliers by step S42. Combined with the judgment criteria of the outlier coefficient, outliers that do not conform to the actual distribution can be scientifically screened and removed. This method avoids the phenomenon of excessive retention or excessive deletion of noise samples in traditional outlier processing methods, making the final abnormal group distribution closer to the real data distribution, and providing a high-quality sample basis for subsequent abnormal detection and analysis. The local reachable density calculation refines the density relationship between the sample and its neighborhood. This interpretability provides a guarantee for the transparency and credibility of the model in practical applications, making it easier for users to understand the source and characteristics of abnormal samples, thereby more effectively evaluating the results of abnormal detection. By supplementing the outlier analysis of abnormal samples, screening out noise samples, and dynamically optimizing the distribution structure of abnormal groups, this method further achieves efficient control of sample quality during abnormal data enhancement, thereby improving the accuracy and robustness of the anomaly detection system.

[0110] Example 5

[0111] Please refer to Figure 1 Specifically: S4 includes the following specific steps:

[0112] S43. On the basis of screening out outliers, re-count the abnormal groups and analyze the balance of the entire sample distribution after the sample enhancement operation to obtain the balance degree Phd. The balance degree Phd is obtained by the following formula:

[0113] ;

[0114] In the formula, Zcs represents the number of samples in the normal group, and Ycs represents the number of samples in the abnormal group.

[0115] The specific steps of S5 include:

[0116] S51, pre-set a balance threshold K, and compare and analyze the balance degree Phd with the balance threshold K to measure the current balance state of the entire sample distribution. The specific measurement content is as follows:

[0117] S511. If the balance degree Phd exceeds the balance threshold K, the entire sample distribution is considered to be not in a balanced state and a red warning is issued;

[0118] S512, if the balance degree Phd does not exceed the balance threshold K, the entire sample distribution is measured to be in a balanced state and a green warning is issued;

[0119] S52. Based on the measurement results in S51, when a green warning is received, there is no need to perform iterative enhancement operations, and the abnormal group will be input into the cloud platform; when a red warning is received, steps S3 and S4 will be repeated to perform iterative enhancement operations until a green warning is received, at which time the iterative enhancement operations are stopped and the abnormal group is input into the cloud platform.

[0120] In this embodiment, by re-stating the data distribution of the abnormal group and the normal group, the method effectively identifies the potential distribution imbalance problem after sample enhancement. The evaluation index of the balance degree Phd can clearly quantify the ratio between normal samples and abnormal samples, guide the enhancement process to further optimize the data structure, avoid sparse or over-dense abnormal data, and improve the overall quality of the data. In step S5, the balance degree is compared and analyzed using a preset balance threshold, and iterative optimization is achieved through dynamic adjustment. This iterative optimization mechanism based on dynamic evaluation can flexibly adjust the sample enhancement strategy according to the changes in real-time data distribution to ensure that the final data distribution reaches a relatively balanced state. At the same time, the method further avoids excessive enhancement or sample waste, and also reduces false positives and false negatives in abnormal detection. By introducing an early warning mechanism: red warning: reminding that the current data distribution has not reached a balanced state and needs to continue to be optimized; green warning: prompting that the current data distribution has reached an ideal state and the enhancement task is completed, this automated judgment and early warning mechanism reduces the need for manual intervention as much as possible and improves detection efficiency. At the same time, the method can issue a warning in time and dynamically adjust the strategy when the data distribution is seriously unbalanced to ensure the stability and reliability of abnormal detection. After dynamic enhancement and optimization, the anomaly group is input into the cloud platform, providing high-quality input data for subsequent data analysis and model training, thereby achieving seamless connection between the detection system and the cloud analysis platform, and providing a foundation for distributed data management and remote monitoring. This cloud-based system architecture not only improves the synergy of data analysis, but also enhances the system's adaptability to large-scale, multi-source data.

[0121] Example 6

[0122] Please refer to Figure 2 ,Specifically: An abnormal data monitoring system based on artificial intelligence, including a data acquisition module, an identification module, a supplement module, an analysis and processing module and an intelligent feedback module;

[0123] The data acquisition module is used to monitor data from different sources in real time using monitoring equipment to generate an original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ;

[0124] The recognition module is used to obtain a standardized feature matrix , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within

[0125] The supplementary module is used to use artificial intelligence technology, combined with the identification content in the identification module, to perform sample enhancement operations on abnormal samples, and introduce distribution constraints during the sample enhancement process to obtain several groups of supplementary abnormal samples , and combined with the identified abnormal samples, generate an abnormal group;

[0126] The analysis and processing module analyzes several groups of supplementary abnormal samples based on the abnormal group. The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd;

[0127] The intelligent feedback module is used to pre-set a balance threshold K and compare and analyze it with the balance degree Phd to measure the current balance state of the entire sample distribution, and based on the measurement results, determine whether it is necessary to perform iterative enhancement operations.

[0128] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An abnormal data monitoring method based on artificial intelligence, characterized in that: The following steps are included: S1. Use monitoring equipment to monitor data from different sources in real time to generate the original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ; S2, based on the standardized feature matrix obtained in S1 , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within The specific steps of S2 include: S21, based on the standardized feature matrix obtained in S12 , analyze the standardized feature matrix The deviation of each sample relative to the entire sample distribution is calculated to obtain the distribution deviation factor Fpyz, which is obtained by the following formula: ； In the formula, represents the distribution deviation factor of the i-th sample; m represents the number of characteristic parameters in the corresponding sample, j=1, 2, 3, ..., m, represents the value of the jth characteristic parameter in the i-th sample, Represents the standardized feature matrix The mean value of the jth characteristic parameter in Represents the standardized feature matrix The standard deviation of the jth feature parameter, i represents the standardized feature matrix The number of samples in ; S3: Using artificial intelligence technology, combined with the identification content in S2, perform sample enhancement on abnormal samples, and introduce distribution constraints during sample enhancement to obtain several groups of supplementary abnormal samples. , and combined with the abnormal samples identified in S2, an abnormal group is generated; S4. Based on the abnormal group, analyze several groups of supplementary abnormal samples The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd; S43. On the basis of screening out outliers, re-count the abnormal groups and analyze the balance of the entire sample distribution after the sample enhancement operation to obtain the balance degree Phd. The balance degree Phd is obtained by the following formula: ； In the formula, Zcs represents the number of samples in the normal group, and Ycs represents the number of samples in the abnormal group; S5. Preset a balance threshold K and compare it with the balance degree Phd to measure the balance state of the entire current sample distribution, and determine whether it is necessary to perform iterative enhancement operations based on the measurement results; The specific steps of S5 include: S51, pre-set a balance threshold K, and compare and analyze the balance degree Phd with the balance threshold K to measure the current balance state of the entire sample distribution. The specific measurement content is as follows: S511. If the balance degree Phd exceeds the balance threshold K, the entire sample distribution is considered to be not in a balanced state and a red warning is issued; S512, if the balance degree Phd does not exceed the balance threshold K, the entire sample distribution is measured to be in a balanced state and a green warning is issued; S52. Based on the measurement results in S51, when a green warning is received, it is not necessary to perform iterative enhancement operations, and the abnormal group is input to the cloud platform; When a red warning is received, steps S3 and S4 are repeated to perform iterative enhancement operations until a green warning is received, at which time the iterative enhancement operations are stopped and the abnormal group is input into the cloud platform.

2. The abnormal data monitoring method based on artificial intelligence according to claim 1 is characterized in that: The specific steps of S1 include: S11. Monitor and record data from different sources to generate the original feature matrix , where, from different sources means from different monitoring devices; S12, the original feature matrix in S11 Perform data preprocessing, where data preprocessing includes the original feature matrix All feature parameters in the data are normalized and standardized, and the feature parameters after data preprocessing are summarized to construct a standardized feature matrix , where the standardized feature matrix Includes each characteristic parameter in each sample.

3. The abnormal data monitoring method based on artificial intelligence according to claim 2 is characterized in that: The specific steps of S2 also include: S22, using the chi-square distribution table, pre-set the critical value P, and compare it with the distribution deviation factor Fpyz to identify the standardized feature matrix The specific identification contents of abnormal samples are as follows: S221, if the distribution deviation factor Fpyz exceeds the critical value P, then the current normalized feature matrix is ​​determined The corresponding samples are in an abnormal state and are marked as abnormal samples; S222: If the distribution deviation factor Fpyz does not exceed the critical value P, the current normalized feature matrix is ​​determined. The corresponding samples are not in abnormal state and are marked as normal samples.

4. The abnormal data monitoring method based on artificial intelligence according to claim 3 is characterized in that: The specific steps of S3 include: S31, according to the method of identifying abnormal samples in S22, several groups of abnormal samples are counted, and the groups of abnormal samples are sorted according to the numerical value of the distribution deviation factor Fpyz, and the abnormal sample with the largest distribution deviation factor Fpyz is extracted as the enhanced reference sample By performing sample enhancement operations on several groups of abnormal samples and introducing distribution constraints during the sample enhancement process, we can obtain additional abnormal samples. , the supplementary abnormal samples The specific formula is as follows: ； In the formula, represents the enhanced benchmark sample; Represents the enhanced benchmark sample A sample randomly selected from the neighborhood of Represents the distribution center of all abnormal samples; Represents a random number; represents the regularization coefficient; Represents the enhanced benchmark sample To the distribution center of all abnormal samples The square distance indicates the degree to which the sample deviates from the center of the distribution; represents the offset in the neighborhood direction; k and g both represent a sample in several groups of abnormal samples; S32, by repeated random sampling and To generate several groups of supplementary abnormal samples ; S33, by respectively obtaining a plurality of groups of abnormal samples and a plurality of groups of supplementary abnormal samples in S31 and S32 Statistics are performed to generate an abnormal group, and statistics are performed on the normal samples marked in S222 to generate a normal group.

5. The abnormal data monitoring method based on artificial intelligence according to claim 4 is characterized in that: The specific steps of S4 include: S41. Analyze several groups of supplementary abnormal samples based on the abnormal group The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The out-of-group coefficient Lqyz is obtained as follows: ； In the formula, represents the outlier coefficient of the a-th supplementary abnormal sample, represents the bth abnormal sample in the neighbor set of the ath supplementary abnormal sample, represents the neighbor set of the a-th supplementary abnormal sample, represents the ath supplementary abnormal sample, represents the local reachable density of the a-th supplementary abnormal sample, represents the number of abnormal samples in the neighbor set of the a-th supplementary abnormal sample, a represents the number of the supplementary abnormal sample, and b represents the number of the abnormal sample in the neighbor set of the a-th supplementary abnormal sample; express The local reachable density reflects The density level within the neighbor set.

6. The abnormal data monitoring method based on artificial intelligence according to claim 5 is characterized in that: The specific steps of S4 also include: S42. Based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter outliers in the abnormal group. The specific contents are as follows: S421, if the out-of-group coefficient Lqyz of the corresponding supplementary abnormal sample is ≥ 1, then determine the corresponding supplementary abnormal sample is an outlier, and the additional abnormal sample Remove from the outlier group; S422: If the out-of-group coefficient Lqyz of the corresponding supplementary abnormal sample is less than 1, then the corresponding supplementary abnormal sample is determined. is not an outlier, and the additional abnormal sample considered as abnormal samples.

7. An abnormal data monitoring system based on artificial intelligence, used to implement the abnormal data monitoring method based on artificial intelligence as described in any one of claims 1 to 6 above, characterized in that: It includes data acquisition module, identification module, supplement module, analysis and processing module and intelligent feedback module; The data acquisition module is used to monitor data from different sources in real time using monitoring equipment to generate an original feature matrix , by taking the original feature matrix Perform data preprocessing to construct a standardized feature matrix ; The recognition module is used to obtain a standardized feature matrix , analyze the standardized feature matrix The deviation of each sample in the sample relative to the entire sample distribution is used to obtain the distribution deviation factor Fpyz. Based on the distribution deviation factor Fpyz value, the standardized feature matrix is ​​identified. Abnormal samples within The supplementary module is used to use artificial intelligence technology, combined with the identification content in the identification module, to perform sample enhancement operations on abnormal samples, and introduce distribution constraints during the sample enhancement process to obtain several groups of supplementary abnormal samples , and combined with the identified abnormal samples, generate an abnormal group; The analysis and processing module analyzes several groups of supplementary abnormal samples based on the abnormal group. The degree of outlier in the abnormal group to obtain corresponding supplementary abnormal samples The outlier coefficient Lqyz is based on the corresponding supplementary abnormal samples The outlier coefficient Lqyz is used to identify and filter out the outliers in the abnormal group, and on the basis of filtering out the outliers, the balance of the entire sample distribution after the sample enhancement operation is analyzed to obtain the balance degree Phd; The intelligent feedback module is used to pre-set a balance threshold K and compare and analyze it with the balance degree Phd to measure the current balance state of the entire sample distribution, and based on the measurement results, determine whether it is necessary to perform iterative enhancement operations.

Images