A cloud computing-based big data anomaly detection method and system

Abstract

The application provides a kind of big data anomaly detection method and system based on cloud computing, it is related to big data anomaly detection technical field, the present application effectively captures complete time sequence mode by business cycle adaptive sliding window segmentation mechanism, and significantly enhances the regularity and continuity of data representation;Deeply integrate multi-dimensional statistical features and entity semantic features driven by graph database to construct a joint volatility quantization model, which greatly improves the robustness of high-dimensional feature space representation and noise immunity;Adopt mutual information dynamic threshold screening strategy combined with gradient boosting decision tree feature stability collaborative optimization mechanism to form a decision paradigm with enhanced noise immunity, thereby achieving accurate identification of abnormal signals and efficient suppression of false alarms in complex dynamic environments;Based on the root cause positioning system of decision path reconstruction and weighted contribution accumulation, the bias problem of traditional feature contribution evaluation is solved, and the identification accuracy and interpretability of key abnormal driving factors are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data anomaly detection, and particularly relates to a big data anomaly detection method and system based on cloud computing. BACKGROUND

[0002] The current technical evolution in the field of big data anomaly detection presents a diversified development trend. With the continuous expansion of the data scale carried by the cloud computing platform, the mainstream scheme generally adopts a sliding time window for time series segmentation, combines statistical feature analysis and entity relationship mining based on a graph database to construct a multi-dimensional feature space; the gradient boosting decision tree becomes a commonly used classifier due to its high efficiency, and mutual information feature screening is also widely used in practice. However, industry observations show that the sudden fluctuations of high-frequency streaming data easily cause the problem of insufficient adaptability of window segmentation to business cycles, and the difficulty in quantifying the stability of multi-source heterogeneous features limits the robustness of the model. At the same time, the challenges such as false alarm rate fluctuation and root cause positioning deviation caused by the sensitivity of decision tree algorithm to feature fluctuations continue to attract attention in the fields of financial real-time risk control and industrial internet operation and maintenance. How to coordinate feature engineering and model decision-making to adapt to complex dynamic environments has become a technical difficulty to be broken through in this field.

[0003] Therefore, it is necessary to provide a big data anomaly detection method and system based on cloud computing to solve the above technical problems. SUMMARY

[0004] To solve the above technical problems, the present application provides a big data anomaly detection method and system based on cloud computing, which achieves the beneficial effect of accurate and effective anomaly data detection.

[0005] The present application provides a big data anomaly detection method based on cloud computing, comprising:

[0006] S1: dividing the preprocessed original data stream into a plurality of continuous standard data segments according to the sliding time window determined by the business cycle;

[0007] S2: extracting the multi-dimensional statistical features and entity identification fields of each standard data segment, and retrieving the pre-constructed graph database according to the entity identification fields to obtain the semantic features of each standard data segment in the preset dimension;

[0008] S3: combining the statistical features and semantic features of the continuous multiple standard data segments into a statistical feature matrix and a semantic feature matrix in time sequence respectively, and calculating a feature fluctuation coefficient vector based on the statistical feature matrix and the semantic feature matrix;

[0009] S4: fusing the statistical feature matrix, the semantic feature matrix and the feature fluctuation coefficient vector to generate a three-dimensional feature tensor;

[0010] S5: Calculate the mutual information value of each feature channel in the three-dimensional feature tensor and the abnormal label in the pre-stored reference label set, construct a dynamic screening threshold according to the statistical characteristics of the mutual information values of all feature channels, and screen the feature channels higher than the dynamic screening threshold to obtain a high-value feature set;

[0011] S6: Input the high-value feature set into the pre-trained gradient boosting decision tree model, obtain an anomaly probability value and a feature heat map by dynamically weighting the feature contribution degree of the feature fluctuation coefficient vector, and output an anomaly detection report according to the anomaly probability value and the feature heat map.

[0012] Preferably, in step S1, the sliding time window length is greater than or equal to 3 complete business cycles.

[0013] Preferably, the preprocessing of the original data stream in step S1 includes the following steps:

[0014] The original data stream is timestamp-aligned to obtain a time-aligned data stream, and the time-aligned data stream is segmented according to a pre-set temporary segmentation window to obtain a plurality of temporary data segments;

[0015] The upper threshold and the lower threshold of each temporary data segment are calculated, and a dynamic boundary interval of each temporary data segment is constructed based on the lower threshold and the upper threshold, wherein the calculation formula of the upper threshold and the lower threshold is:

[0016] ;

[0017] wherein, is the upper threshold, is the third quartile of the temporary data segment, is the first quartile of the temporary data segment, is an empirical scaling coefficient, is the interquartile range, is a dynamic expansion coefficient, is the standard deviation of the temporary data segment;

[0018] The dynamic boundary interval is constructed based on the lower threshold and the upper threshold, the abnormal data points in each temporary data segment that are outside the dynamic boundary interval are removed, and the temporary data segments after removing the abnormal data points are merged in time stamp order to generate a standardized data stream.

[0019] Preferably, in step S3, the list in the statistical feature matrix and the semantic feature matrix represents the values of all standard data segments of one feature dimension.

[0020] Preferably, in step S3, the generation step of the feature fluctuation coefficient vector includes:

[0021] The coefficient of variation is calculated for each column of data in the statistical feature matrix, and the standard deviation is calculated for each column of data in the semantic feature matrix.

[0022] Preferably, in step S5, the construction of the dynamic screening threshold comprises the following steps:

[0023] The average value and the standard deviation of the mutual information values of all feature channels are calculated, and the value of the average value plus three times the standard deviation is obtained to obtain a first candidate threshold value;

[0024] The proportion value of the number of abnormal labels in the pre-stored reference label set to the total number of labels in the reference label set is calculated, and the natural logarithm of the reciprocal of the proportion value is obtained to obtain a second candidate threshold value;

[0025] The larger value of the first candidate threshold value and the second candidate threshold value is taken as the dynamic screening threshold.

[0026] Preferably, in step S6, inputting the high-value feature set into the pre-trained gradient boosting decision tree model further comprises:

[0027] The feature fluctuation coefficient is multiplied by the mutual information value to generate a weighted confidence;

[0028] The input order of the high-value feature set is reconstructed in descending order of the weighted confidence.

[0029] Preferably, in step S6, a depth attenuation factor is added to the split gain calculation of the pre-trained gradient boosting decision tree model.

[0030] Preferably, in step S6, the step of obtaining an anomaly probability value and a feature heat map by dynamically weighting feature contribution degree through the feature fluctuation coefficient vector comprises:

[0031] A weight factor is generated through the feature fluctuation coefficient vector, the decision tree node split evaluation value is real-time corrected based on the weight factor to reconstruct the decision path, the anomaly probability value is output based on the terminal node distribution of the reconstructed decision path, and the feature heat map is formed by accumulating the path split contribution.

[0032] The application also provides a big data anomaly detection system based on cloud computing, which is applied to a big data anomaly detection method based on cloud computing and comprises:

[0033] A data segmentation module is configured to divide the preprocessed original data stream into a plurality of continuous standard data segments according to a sliding time window determined by a business cycle.

[0034] A feature joint extraction module is configured to extract multi-dimensional statistical features and entity identification fields of each standard data segment, and retrieve a pre-constructed graph database according to the entity identification fields to obtain semantic features of each standard data segment in a preset dimension.

[0035] a fluctuation feature quantification module, configured to combine statistical features and semantic features of a plurality of standard data segments in time sequence into a statistical feature matrix and a semantic feature matrix respectively, and calculate a feature fluctuation coefficient vector based on the statistical feature matrix and the semantic feature matrix;

[0036] a tensor fusion module, configured to fuse the statistical feature matrix, the semantic feature matrix and the feature fluctuation coefficient vector to generate a three-dimensional feature tensor;

[0037] a mutual information dynamic screening module, configured to calculate a mutual information value of each feature channel in the three-dimensional feature tensor and an abnormal label in a pre-stored benchmark label set, construct a dynamic screening threshold according to statistical features of mutual information values of all feature channels, and screen feature channels higher than the dynamic screening threshold to obtain a high-value feature set;

[0038] a noise-resistant decision analysis module, configured to input the high-value feature set into a pre-trained gradient boosting decision tree model, obtain an abnormal probability value and a feature heat map by dynamically weighting feature contribution degrees of the feature fluctuation coefficient vector, and output an abnormal detection report according to the abnormal probability value and the feature heat map.

[0039] Compared with the related art, the method and system for detecting big data anomalies based on cloud computing have the following beneficial effects:

[0040] The application effectively captures complete time sequence patterns through a business cycle adaptive sliding window segmentation mechanism, and significantly enhances the regularity and continuity of data representation; a joint fluctuation quantification model is constructed by deeply fusing multi-dimensional statistical features and entity semantic features driven by a graph database, which greatly improves the representation robustness and noise immunity of a high-dimensional feature space; a noise-resistant enhanced decision paradigm is formed by adopting a mutual information dynamic threshold screening strategy combined with a feature stability collaborative optimization mechanism of a gradient boosting decision tree, so that accurate identification of abnormal signals and efficient suppression of false positives are realized in a complex dynamic environment; a root cause positioning system based on decision path reconstruction and weighted contribution accumulation breaks through the bias problem of feature contribution evaluation in traditional methods, and significantly improves the identification accuracy and interpretability of key abnormal driving factors; the entire technical solution integrates the multi-stage processes of data preprocessing, feature engineering and model decision into an organic and collaborative computing framework, while maintaining real-time processing capability, greatly enhances the generalization adaptability of the system to business scenario migration and sudden abnormal patterns, and realizes big data anomaly detection in cloud computing environment in terms of accuracy, robustness, interpretability and scenario adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 a flowchart of the method for detecting big data anomalies based on cloud computing;

[0042] Figure 2A module structure diagram of a cloud computing-based big data anomaly detection system. DETAILED DESCRIPTION

[0043] The application will be described in further detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application and not to limit the application. In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings, not all the structures. In addition, the embodiments in the application and the features in the embodiments can be combined with each other without conflict.

[0044] In addition, it should be noted that, for the sake of description, only the parts related to the application are shown in the drawings, not all the contents. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes can be terminated when the operations are completed, but can also have additional steps not included in the drawings. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0045] Embodiment one

[0046] A cloud computing-based big data anomaly detection method, in the specific implementation process, as shown in Figure 1 , which shows a flowchart of a cloud computing-based big data anomaly detection method, including:

[0047] Step S1: The preprocessed original data stream is divided into a plurality of continuous standard data segments according to the sliding time window determined by the business cycle.

[0048] Specifically, in step S1, the sliding time window length is greater than or equal to 3 complete business cycles.

[0049] Specifically, the preprocessing of the original data stream in step S1 includes the following steps:

[0050] The time alignment data stream is obtained by timestamp alignment of the original data stream, and the time alignment data stream is divided into a plurality of temporary data segments according to the preset temporary division window.

[0051] The upper threshold and the lower threshold of each temporary data segment are calculated, and the dynamic boundary interval of each temporary data segment is constructed based on the lower threshold and the upper threshold, wherein the calculation formula of the upper threshold and the lower threshold is:

[0052] ;

[0053] wherein, is an upper bound threshold, is a third quartile of the temporary data segment, is a first quartile of the temporary data segment, is an empirical scaling coefficient, is a quartile range, is a dynamic expansion coefficient, is a standard deviation of the temporary data segment;

[0054] a dynamic boundary interval is constructed based on the lower bound threshold and the upper bound threshold, abnormal data points exceeding the dynamic boundary interval in each temporary data segment are removed, and the temporary data segments after the abnormal data points are removed are merged in chronological order to generate a standardized data stream.

[0055] In the specific implementation process, first, the original data stream is time-stamped to the millisecond level by the network time protocol to eliminate cross-source time sequence deviation and generate time-aligned data stream, and then based on a preset temporary segmentation window, for example, the length is usually 1 / 10 to 1 / 5 of the business cycle, the time-aligned data stream is cut into continuous temporary data segments, and a dynamic boundary construction operation is performed on each temporary data segment: the first quartile of the data segment is calculated and the third quartile is and the quartile range is obtained , the standard deviation of the data segment is calculated synchronously , the threshold value is made data adaptive by substituting the dynamic expansion coefficient formula , the standard deviation is nonlinearly increased to contain reasonable fluctuations, and then the upper bound threshold and the lower bound threshold are calculated according to the empirical scaling coefficient , for example, the typical value is 1.5, forming a dynamic boundary interval that wraps the main body of the data, discrete data points outside the dynamic boundary interval are regarded as outliers and are removed, and finally the effective data of all temporary data segments after denoising are reassembled in chronological order to generate a standardized data stream; for this standardized data stream, according to the business cycle characteristics, for example, financial transactions have a 24-hour cycle, and industrial equipment has an 8-hour production shift cycle, the length of the sliding window is determined, which is forced to be greater than or equal to 3 complete business cycles, and a number of standard data segments are generated by sliding interception with a fixed step length as the input basis for subsequent feature processing. Through the dynamic stretching mechanism of the temporary window and the period adaptation mechanism of the business window, both transient noise interference and complete retention of cross-cycle regularity patterns are achieved, which can effectively retain the periodic characteristics of the data while reducing the proportion of invalid data segments caused by network jitter in the cloud computing environment.

[0056] Step S2: extract the multi-dimensional statistical features and entity identification field of each standard data segment, and retrieve the pre-constructed graph database according to the entity identification field to obtain the preset dimension semantic features of each standard data segment.

[0057] In the implementation process, first, perform a multi-dimensional statistical feature extraction operation on each standard data segment, including but not limited to time domain mean, variance, skewness, and frequency domain wavelet energy proportion, and other multi-dimensional basic statistical indicators, while analyzing the data packet header information including but not limited to obtaining the device code, operator ID, and other entity identification fields; based on this entity identification field, activate the pre-constructed graph database retrieval process, the construction of the pre-constructed graph database includes collecting business full-chain entity relationship data, for example, device number, sensor node, process link in industrial scenarios, account ID, counterparty, and fund flow direction in financial scenarios, convert entities into graph nodes and assign static attributes such as device model and account level, and convert the interaction relationship between entities into weighted directed edges, for example, device data stream transmission frequency and account fund flow amount, use Neo4j engine to build a distributed graph structure storage cluster, node attributes include time effectiveness labels, for example, device remaining life estimation update timestamp, and edge attribute embedded probabilistic transmission quality parameters, achieve 1 billion node millisecond response; after completing real-time retrieval of the graph database, output the dynamic semantic feature vector strongly associated with the entity identification, and finally form a mixed feature containing numerical statistical features and associated semantic features.

[0058] Step S3: combine the statistical features and semantic features of the continuous multiple standard data segments into a statistical feature matrix and a semantic feature matrix in time sequence, respectively, and calculate a feature fluctuation coefficient vector based on the statistical feature matrix and the semantic feature matrix.

[0059] Specifically, in step S3, the list in the statistical feature matrix and the semantic feature matrix represents the values of all standard data segments of one feature dimension.

[0060] Specifically, in step S3, the generation step of the feature fluctuation coefficient vector includes:

[0061] Calculate the coefficient of variation for each column of data in the statistical feature matrix, and calculate the standard deviation for each column of data in the semantic feature matrix, and concatenate the coefficient of variation and the standard deviation in dimension order to form the feature fluctuation coefficient vector.

[0062] In the implementation process, first, a plurality of continuous standard data segments are sorted in time sequence, and data structures are constructed for statistical feature dimensions and semantic feature dimensions. The column vector of the statistical feature matrix represents the value sequence of a specific statistical feature on all data segments, denoted as a statistical feature column. The column vector of the semantic feature matrix represents the value sequence of a preset dimension semantic feature on all data segments, denoted as a semantic feature column. The key technical constraint is that the values of all data segments in each feature dimension must be strictly arranged in time stamp order and cannot be out of position. Then, the feature fluctuation coefficient is calculated. For the statistical feature column, the dimensionless coefficient of variation is used to calculate the value of the standard deviation divided by the mean value of the column to eliminate the influence of dimensional difference on the evaluation of statistical feature stability. For the semantic feature column, the standard deviation of the semantic feature is calculated to retain the original dimensional fluctuation information and reflect the semantic shift degree. Finally, the feature fluctuation coefficient vector is constructed in the order of feature dimensions. The splicing rule requires that the front segment element is the statistical feature coefficient of variation arranged in the order of feature declaration, and the rear segment element is the standard deviation of the semantic feature arranged in the order of graph database dimension definition, which provides a basis for subsequent steps.

[0063] Step S4: The statistical feature matrix, the semantic feature matrix, and the feature fluctuation coefficient vector are fused to generate a three-dimensional feature tensor.

[0064] In the implementation process, first, the statistical feature matrix and the semantic feature matrix are dimensionally aligned. The two matrices are uniformly expanded to the same feature dimension through zero padding to form an equal dimension matrix. At the same time, the feature fluctuation coefficient vector is copied multiple times along the time dimension through the longitudinal broadcast mechanism to generate a fluctuation coefficient matrix. The number of copies is strictly equal to the total number of time windows. Then, a three-channel splicing operation is performed along the channel axis: the first channel fills the zero-extended statistical feature matrix, the second channel fills the zero-extended semantic feature matrix, and the third channel fills the broadcast-extended fluctuation coefficient matrix. Finally, a three-dimensional feature tensor is generated. Through tensor fusion, static features and dynamic stability indicators are established in a unified high-dimensional space to establish a spatio-temporal correlation, so that the subsequent mutual information calculation can simultaneously capture the feature value distribution rule and the stability mode across time periods.

[0065] Step S5: Calculate the mutual information value of each feature channel in the three-dimensional feature tensor and the pre-stored abnormal label set. According to the statistical characteristics of the mutual information values of all feature channels, a dynamic screening threshold is constructed to screen the feature channels higher than the dynamic screening threshold to obtain a high-value feature set.

[0066] Specifically, in step S5, the construction of the dynamic screening threshold includes the following steps:

[0067] The average value and the standard deviation of the mutual information values of all feature channels are calculated. The value of the average value plus three times the standard deviation is obtained to obtain a first candidate threshold.

[0068] Calculate the proportion of the number of abnormal labels in the pre-stored benchmark label set to the total number of labels in the benchmark label set, and take the natural logarithm of the reciprocal of the proportion value to obtain a second candidate threshold value;

[0069] Take the larger value of the first candidate threshold value and the second candidate threshold value as the dynamic screening threshold value.

[0070] In the implementation process, first, load the pre-stored benchmark label set, which is a multi-dimensional label library containing explicitly labeled abnormal events and normal events constructed through historical data accumulation. The time span needs to cover at least three typical business cycles and the total sample size should not be less than 100,000 records to ensure that the distribution of abnormal labels can reflect the statistical laws of the real scene. Then, for each feature channel of the three-dimensional feature tensor, that is, the data slice representing a single feature dimension in the tensor, mutual information value calculation is performed: form a histogram distribution by discretizing the continuous values of the feature channel, calculate the joint probability density with the abnormal label distribution in the benchmark label set, and finally quantify the statistical correlation strength of the feature channel and the abnormal event based on the Shannon entropy formula. After completing the mutual information value calculation of all channels, enter the dynamic screening threshold value construction stage: first, calculate the arithmetic mean and standard deviation of all mutual information values, and add three times the standard deviation to the mean to obtain the first candidate threshold value, which can cover 99.7% of the conventional feature distribution interval. At the same time, calculate the proportion of the number of abnormal labels in the benchmark label set to the total number of labels, and take the natural logarithm of the reciprocal of the proportion value as the second candidate threshold value. Finally, select the larger value of the first candidate threshold value and the second candidate threshold value as the dynamic screening threshold value. Through the double-threshold mechanism, both high-discrimination features are captured through statistical outliers and low-frequency high-risk events are adapted through the inverse function of abnormal probability; in the screening stage, feature channels with mutual information values exceeding the dynamic screening threshold value are included in the high-value feature set, and invalid channels with mutual information values less than 0.01 are forcibly excluded to avoid noise interference. The generated feature set establishes a mapping relationship with the original feature dimension through hash indexing, providing a noise-resistant input source for subsequent decision tree models.

[0071] Step S6: input the high-value feature set into the pre-trained gradient boosting decision tree model, dynamically weight the feature contribution degree through the feature fluctuation coefficient vector to obtain the abnormal probability value and the feature heat map, and output the abnormal detection report according to the abnormal probability value and the feature heat map.

[0072] Specifically, in step S6, inputting the high-value feature set into the pre-trained gradient boosting decision tree model further includes:

[0073] Multiply the feature fluctuation coefficient and the mutual information value to generate a weighted confidence;

[0074] Reconstruct the input order of the high-value feature set in descending order of the weighted confidence.

[0075] Specifically, in step S6, a depth attenuation factor is added in the split gain calculation of the pre-trained gradient boosting decision tree model.

[0076] Specifically, in step S6, the step of obtaining an anomaly probability value and a feature heat map by dynamically weighting feature contribution degrees through a feature fluctuation coefficient vector includes:

[0077] A weight factor is generated through a feature fluctuation coefficient vector, the evaluation value of the decision tree node splitting is corrected in real time based on the weight factor to reconstruct the decision path, the anomaly probability value is output based on the terminal node distribution of the reconstructed decision path, and the feature heat map is formed by accumulating the path splitting contribution.

[0078] In the implementation process, first, the high-value feature set is pre-processed and strengthened, the stability parameter of each feature is extracted based on the feature fluctuation coefficient vector, and the product operation is performed with the mutual information value of the corresponding feature to generate a weighted confidence score, and then the input sequence of the high-value feature set is reconstructed in descending order of the score, so that the features with high stability and strong correlation are preferentially processed by the model; then the reconstructed feature sequence is input into the pre-trained gradient boosting decision tree model, which is constructed by training historical multi-source data, and the training process includes three stages, the first stage selects the standardized abnormal detection data set of the past six months as the basic sample, the second stage uses the approximate second derivative acceleration algorithm of Taylor expansion for iterative optimization of the decision tree split point, and the third stage automatically adjusts the learning rate and tree complexity based on the validation set accuracy after each round of lifting calculation until the model converges to a stable accuracy threshold; in the model execution stage, a double optimization mechanism is implemented simultaneously in the node splitting process: on the one hand, a depth attenuation factor is introduced in the split gain calculation, and the gain weight decreases exponentially with each increase in split depth to suppress overfitting, on the other hand, the stability parameter is indexed according to the current feature identifier by reading the feature fluctuation coefficient vector in real time, and a weight factor is generated through negative correlation mapping, for example, when the fluctuation coefficient is 0.3, the weight factor is 1 / 0.3, which is about 3.33, multiply the weight factor by the benchmark split evaluation value, for example, the benchmark split evaluation value is the Gini index gain or information entropy reduction, to construct a weighted split decision to replace the original split rule and reconstruct the entire decision path; finally, the abnormal probability value is calculated based on the terminal node distribution of the reconstructed decision path in the output layer, and the stability contribution value of each feature is accumulated along the weighted split path, and a visual feature heat map is generated through normalization mapping, the abnormal probability value output by the whole process is used as a real-time risk quantification index to directly drive the automatic response strategy, the feature heat map highlights the physical location of the key root cause feature in the system topology map through the dyeing mapping mechanism, and the structured abnormal detection report integrates the original data segment, the probability value evolution curve, the heat root cause distribution map and the historical similar case disposal suggestion to form a closed-loop decision chain, so that the operation and maintenance personnel can not only confirm the abnormal state in time, but also accurately locate the fault equipment position or attack entry point, and generate executable operation instructions in a short time by referring to the historical strategy template, finally, the abnormal perception, root cause diagnosis and disposal decision in the cloud computing environment are realized in a second-level automatic closed loop, and the whole process reduces the decision interference rate of unstable features and improves the heat prominence of key root cause features through the feature stability perception mechanism.

[0079] The working principle of the big data anomaly detection method based on cloud computing provided by the application is as follows:

[0080] The application generates continuous standard data segments by time stamp alignment and adaptive anomaly filtering mechanism based on quartile range to denoise and standardize the original data stream;On this basis, multi-dimensional statistical features are extracted and combined with graph database semantic features to construct a dual feature matrix, and the variation coefficient is calculated for the statistical feature matrix, and the standard deviation is calculated for the semantic feature matrix to form a joint volatility coefficient vector to represent the feature stability;Then the feature matrix and the volatility coefficient vector are fused into a three-dimensional space-time tensor by zero padding and broadcast expansion, and the high-value feature set is locked by mutual information dynamic threshold screening;When inputting the pre-trained gradient boosting decision tree model, the weighted confidence generated by the product of the volatility coefficient and the mutual information is used to optimize the feature input sequence, and the dual decision intervention is implemented in the model inference process, and the depth attenuation factor is applied in the node splitting stage to suppress overfitting and the dynamic weight factor based on the volatility coefficient to reconstruct the splitting path;Finally, the abnormal probability is calculated according to the terminal node distribution of the weighted decision path to quantify the risk level, and the root cause heat map is generated by backtracking the depth correction contribution value of each feature in the weighted splitting point, both of which drive the automatic operation and maintenance strategy, realize the whole link closed loop from data preprocessing, feature stability quantification, model anti-interference decision to result interpretability output.

[0081] Embodiment two

[0082] A big data anomaly detection system based on cloud computing is applied to a big data anomaly detection method based on cloud computing. Figure 2 As shown in the figure, it shows a module structure diagram of a big data anomaly detection system based on cloud computing, which includes:

[0083] The working principle of the big data anomaly detection system based on cloud computing provided by the application is as follows:

[0084] The data segmentation module is used for segmenting the preprocessed original data stream into multiple continuous standard data segments according to the sliding time window determined by the business cycle.

[0085] The feature joint extraction module is used for extracting the multi-dimensional statistical features and entity identification fields of each standard data segment, and retrieving the pre-constructed graph database according to the entity identification fields to obtain the preset dimension semantic features of each standard data segment.

[0086] The volatility feature quantification module is used for combining the statistical features and semantic features of the continuous multiple standard data segments into statistical feature matrices and semantic feature matrices in time sequence respectively, and calculating the feature volatility coefficient vector based on the statistical feature matrices and the semantic feature matrices.

[0087] The tensor fusion module is used for fusing the statistical feature matrices, the semantic feature matrices and the feature volatility coefficient vector to generate a three-dimensional feature tensor.

[0088] The mutual information dynamic screening module is used for calculating mutual information values of each feature channel in the three-dimensional feature tensor and the pre-stored abnormal label in the benchmark label set, and constructing a dynamic screening threshold according to statistical characteristics of the mutual information values of all the feature channels, and screening the feature channels higher than the dynamic screening threshold to obtain a high-value feature set.

[0089] The anti-interference decision analysis module is used for inputting the high-value feature set into a pre-trained gradient boosting decision tree model, obtaining an abnormal probability value and a feature heat map through dynamic weighting of feature contribution degrees of a feature fluctuation coefficient vector, and outputting an abnormal detection report according to the abnormal probability value and the feature heat map.

[0090] The working principle of the big data abnormal detection system based on cloud computing provided by the application is as follows:

[0091] The application generates standard data segments with complete patterns according to business cycles through the streaming data segmentation module 100, analyzes multi-dimensional statistical features and graph database semantic features to form a dual-modal space-time matrix through the feature joint extraction module 200, performs cross-window coefficient of variation and standard deviation calculation on the matrix to build a global feature fluctuation coefficient vector through the fluctuation feature quantization module 300, splices the feature matrix and the fluctuation vector after zero padding and broadcast expansion into a three-dimensional feature tensor along the channel axis through the tensor fusion module 400, calculates the abnormal correlation degree of each channel according to the three-dimensional feature tensor through the mutual information dynamic screening module 500, and selects a high-value feature set through a double-threshold optimization mechanism based on statistical distribution and information theory risk; the anti-interference decision analysis module 600 takes the feature fluctuation coefficient as a weight factor to reconstruct the splitting path decision rule of the gradient boosting decision tree, combines depth attenuation to suppress deep node overfitting interference, dynamically weights and strengthens the decision weight of stable features, finally generates an abnormal probability risk value according to the weighted path terminal distribution, and forms a root cause heat map by backtracking and accumulating the depth correction contribution along the splitting node, so as to realize a full-link closed-loop processing framework from dynamic feature stability quantization, model anti-interference decision optimization to interpretable result output.

[0092] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device realize the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for realizing the functions specified in one flow or multiple flows and / or blocks

[0093] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk storage, magnetic disk storage, magnetic tape storage, or any other medium that can be used to carry or store data which can be read by a computer.

[0094] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the statement "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

Claims

1. A big data anomaly detection method based on cloud computing, characterized in that, The anomaly detection method includes the following steps: S1: Divide the preprocessed raw data stream into multiple consecutive standard data segments according to the sliding time window determined by the business cycle; S2: Extract the multidimensional statistical features and entity identifier fields of each standard data segment, and retrieve the semantic features of each standard data segment in the preset dimension based on the entity identifier fields from the pre-built graph database; S3: Combine the statistical and semantic features of multiple consecutive standard data segments into statistical feature matrices and semantic feature matrices in chronological order, and calculate the feature fluctuation coefficient vector based on the statistical feature matrices and semantic feature matrices; S4: Fuse the statistical feature matrix, semantic feature matrix, and feature fluctuation coefficient vector to generate a three-dimensional feature tensor; S5: Calculate the mutual information value between each feature channel in the three-dimensional feature tensor and the abnormal labels in the pre-stored benchmark label set. Construct a dynamic screening threshold based on the statistical characteristics of the mutual information values ​​of all feature channels, and select feature channels that are higher than the dynamic screening threshold to obtain a high-value feature set. S6: Input the high-value feature set into the pre-trained gradient boosting decision tree model, obtain the anomaly probability value and feature heatmap by dynamically weighting the feature contribution through the feature fluctuation coefficient vector, and output an anomaly detection report based on the anomaly probability value and feature heatmap. In step S5, the construction of the dynamic filtering threshold includes the following steps: The first candidate threshold is obtained by calculating the average and standard deviation of the mutual information values ​​of all feature channels, plus three times the standard deviation. Calculate the ratio of the number of abnormal tags in the pre-stored baseline tag set to the total number of tags in the baseline tag set, and take the natural logarithm of the reciprocal of the ratio to obtain the second candidate threshold. The larger value between the first candidate threshold and the second candidate threshold is taken as the dynamic filtering threshold; In step S6, inputting the high-value feature set into the pre-trained gradient boosting decision tree model further includes: The weighted confidence level is generated by multiplying the characteristic fluctuation coefficient with the mutual information value. Reconstruct the input order of the high-value feature set in descending order of weighted confidence; In step S6, a depth decay factor is added to the split gain calculation of the pre-trained gradient boosting decision tree model; In step S6, the steps of obtaining the anomaly probability value and the feature heatmap by dynamically weighting the feature contribution degree through the feature fluctuation coefficient vector include: Weighting factors are generated by using feature fluctuation coefficient vectors. The decision tree node splitting evaluation values ​​are then corrected in real time based on the weighting factors to reconstruct the decision path. Anomaly probability values ​​are output based on the distribution of terminal nodes in the reconstructed decision path, and the path splitting contribution is accumulated to form a feature heatmap.

2. The big data anomaly detection method based on cloud computing according to claim 1, characterized in that, In step S1, the length of the sliding time window is greater than or equal to three complete business cycles.

3. The big data anomaly detection method based on cloud computing according to claim 2, characterized in that, The preprocessing of the raw data stream in step S1 includes the following steps: The original data stream is timestamped to obtain a time-aligned data stream, and the time-aligned data stream is divided into multiple temporary data segments according to a preset temporary segmentation window; Calculate the upper and lower bound thresholds for each temporary data segment, and construct the dynamic boundary interval for each temporary data segment based on the lower and upper bound thresholds. The formulas for calculating the upper and lower bound thresholds are as follows: ； in, This is the upper threshold. The third quartile of the temporary data segment. The first quartile of the temporary data segment. This is an empirical scaling factor. Interquartile range, This is the dynamic expansion coefficient. The standard deviation of the temporary data segment; A dynamic boundary interval is constructed based on the lower and upper bound thresholds. Abnormal data points that exceed the dynamic boundary interval in each temporary data segment are removed. The temporary data segments after removing abnormal data points are merged in timestamp order to generate a standardized data stream.

4. The big data anomaly detection method based on cloud computing according to claim 3, characterized in that, In step S3, the columns in the statistical feature matrix and the semantic feature matrix represent the values ​​of all standard data segments of a feature dimension.

5. The big data anomaly detection method based on cloud computing according to claim 4, characterized in that, In step S3, the generation of the characteristic fluctuation coefficient vector includes: Calculate the coefficient of variation for each column of the statistical feature matrix and the standard deviation for each column of the semantic feature matrix. Concatenate the coefficient of variation and the standard deviation in dimensional order to form a feature fluctuation coefficient vector.

6. A big data anomaly detection system based on cloud computing, characterized in that, The anomaly detection system, applicable to any one of claims 1 to 5, is a cloud-based big data anomaly detection method, comprising: The data segmentation module is used to divide the preprocessed raw data stream into multiple continuous standard data segments according to the sliding time window determined by the business cycle; The feature joint extraction module is used to extract multidimensional statistical features and entity identification fields for each standard data segment, and retrieve the semantic features of each standard data segment in a preset dimension based on the entity identification field from the pre-built graph database. The fluctuation feature quantization module is used to combine the statistical and semantic features of multiple consecutive standard data segments into statistical feature matrices and semantic feature matrices in chronological order, and calculate the feature fluctuation coefficient vector based on the statistical feature matrices and semantic feature matrices. The tensor fusion module is used to fuse statistical feature matrices, semantic feature matrices, and feature fluctuation coefficient vectors to generate three-dimensional feature tensors. The mutual information dynamic filtering module is used to calculate the mutual information value between each feature channel in the 3D feature tensor and the abnormal labels in the pre-stored benchmark label set. Based on the statistical characteristics of the mutual information values ​​of all feature channels, a dynamic filtering threshold is constructed, and feature channels with values ​​higher than the dynamic filtering threshold are filtered to obtain a high-value feature set. The anti-disturbance decision parsing module is used to input the high-value feature set into the pre-trained gradient boosting decision tree model, obtain the anomaly probability value and feature heatmap by dynamically weighting the feature contribution through the feature fluctuation coefficient vector, and output an anomaly detection report based on the anomaly probability value and feature heatmap.

Images