Big data anomaly detection system based on deep learning

Abstract

The application discloses a big data anomaly detection system based on deep learning and relates to the technical field of deep learning.The system comprises a data collection and analysis module, an unconventional mode detection module, a mode differentiation processing module, an anomaly detection adjustment module and an anomaly early warning module.The application collects multi-source behavior data of each access user and analyzes access mode types.In a conventional mode, access behavior data sets are input into a global deep representation model for reconstruction analysis.In a local anomaly mode, behavior features with high reconstruction error feature contribution degree are screened to form a local feature subspace and perform local reconstruction analysis.In the anomaly detection adjustment stage, the input features of the coding layer are re-executed for anomaly detection processing, thereby avoiding the problem that abnormal features are diluted in the deep representation space in a high-dimensional sparse data environment and achieving the effect that abnormal behaviors can be accurately identified in a complex data environment.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, specifically to a big data anomaly detection system based on deep learning. Background Technology

[0002] With the development of information technology, various information systems continuously generate a large amount of operational data during operation. This data is typically characterized by large scale, high feature dimensionality, diverse sources, and dynamic changes over time. Therefore, it is necessary to continuously monitor the system's operational status through data analysis technology in order to identify abnormal behavior or potential faults in a timely manner. Since deep learning technology has powerful automatic feature learning capabilities, it can mine complex nonlinear relationships from high-dimensional data and build normal behavior models without relying on manual feature engineering. Therefore, deep learning is widely used in big data anomaly detection.

[0003] For example, Chinese invention patent CN109766992B discloses a deep learning-based method for industrial control system anomaly detection and attack classification, including a Mahalanobis distance-based feature mapping method for industrial control system flow. This method considers the actual situation of industrial control systems, using Mahalanobis distance between features for correlation measurement, and converts the original one-dimensional flow data into a two-dimensional matrix used as input to a convolutional neural network model. By analyzing the shortcomings of existing anomaly detection methods, a convolutional neural network model is used for detection and classification. Simultaneously, considering the characteristics of the relationships between various features in industrial control systems, a Mahalanobis distance-based feature mapping method is proposed to convert one-dimensional flow data into a two-dimensional matrix used as input to a CNN.

[0004] For example, Chinese invention patent CN115169527B discloses a method for detecting abnormal ship states based on AIS data. This method includes acquiring raw AIS data, preprocessing the raw AIS data to obtain the ship's original trajectory, dividing the original trajectory into segments to obtain straight and curved segments; after extracting the turning and straight segments, using a trajectory similarity metric algorithm to remove abnormal trajectories; and using a Bi-LSTM model to make predictions based on the absence of abnormal trajectories. Finally, based on the straight and curved segments, a prediction-based anomaly detection model is constructed using a deep learning model (Bi-LSTM) to complete the detection of abnormal ship states.

[0005] However, in real-world big data environments, due to the extreme scarcity of sudden, abnormal, and unknown types of anomalies, and the fact that their corresponding features account for a very small percentage in the high-dimensional sparse space, the model will prioritize fitting the dominant specific data patterns during training. This causes the abnormal features to be gradually diluted in the deep representation space, making it impossible to form a significant discriminative representation. Ultimately, this leads to an increase in the false negative rate and failure of early warnings, severely limiting the actual effectiveness of deep learning in big data anomaly detection. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a big data anomaly detection system based on deep learning.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] This invention discloses a data acquisition and analysis module for collecting multi-source behavioral data from various users and analyzing the access mode type of the current time window, including regular and non-regular modes; an extraordinary detection module for performing behavioral feature analysis and filtering out sudden and local anomaly modes if the current time window is extraordinary, otherwise not performing behavioral feature analysis; a mode differentiation processing module for performing corresponding data anomaly detection processing on regular, sudden, and local anomaly modes respectively, obtaining corresponding data anomaly detection processing results, and statistically obtaining a comprehensive anomaly detection result, including regular anomaly detection results, sudden anomaly detection results, and local anomaly detection results, and the comprehensive anomaly detection result including detected anomaly and detected normal; an anomaly detection adjustment module for performing differentiated parameter adjustment and detection optimization processing based on the data anomaly detection processing result if the comprehensive anomaly detection result is detected as anomaly, and re-performing anomaly detection based on the optimized parameters to obtain optimized detection results, otherwise not performing processing; and an anomaly early warning module for confirming that the current time window is a real anomaly state and triggering a corresponding early warning signal when the optimized detection result is unqualified, otherwise not triggering an early warning.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0010] 1. This solution collects multi-source behavioral data from various users and analyzes it to obtain access pattern types. Under normal mode, the access behavior dataset is input into the global deep representation model for reconstruction analysis. Simultaneously, under local anomaly mode, behavioral features with high contribution to reconstruction error features are selected to form a local feature subspace and local reconstruction analysis is performed. In the anomaly detection adjustment stage, anomaly detection processing is re-executed on the input features of the encoding layer. This enhances the expressive power of anomalous behavioral features in the representation space during the deep representation process, thereby avoiding the problem of anomalous features being diluted in the deep representation space under high-dimensional sparse data environment. This achieves the effect of accurately identifying anomalous behavior even in complex data environments.

[0011] 2. This solution obtains an access behavior dataset by performing time window aggregation analysis on multi-source behavioral data. Based on the deviation magnitude and rate of change of the access behavior data, it determines the access pattern type. When the access pattern is identified as non-severe, it is further screened by combining the instantaneous resource dataset to distinguish between burst patterns and local anomaly patterns. For different access patterns, global deep representation model analysis, time window adjustment processing, and local feature subspace modeling processing are performed respectively. This enables the anomaly detection process to differentiate the processing of different types of access behavior changes, thereby avoiding misjudging normal burst access behavior as abnormal access behavior. At the same time, it can identify local anomalies that occur only in a few behavioral features, thus improving the accuracy of the comprehensive anomaly detection results.

[0012] 3. This solution adjusts the time window, extends the burst analysis time interval, and adjusts the encoding compression dimension based on the different data anomaly detection results after obtaining the results of regular anomaly detection, burst anomaly detection, and local anomaly detection. Then, it re-executes the anomaly detection process based on the optimized parameters to obtain optimized detection results. This allows the anomaly detection system to dynamically adjust the detection parameters according to the detection results, thereby enabling the anomaly detection model to maintain appropriate detection granularity and feature expression capabilities under different access behavior changes. This achieves adaptive optimization of the anomaly detection process and improves the stability and detection accuracy of the system in a continuous operating environment. Attached Figure Description

[0013] The disclosure of this invention is illustrated with reference to the accompanying drawings. It should be understood that the drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings, the same reference numerals are used to refer to the same parts. Wherein:

[0014] Figure 1 This is a system architecture diagram of the present invention;

[0015] Figure 2 This is a schematic diagram of the overall process of the present invention;

[0016] Figure 3 This is a flowchart of the encoding compression dimension adjustment and local anomaly optimization detection process of the present invention. Detailed Implementation

[0017] It is readily understood that, based on the technical solution of this invention, those skilled in the art can propose various interchangeable structural methods and implementations without altering the essential spirit of the invention. Therefore, the following detailed embodiments and accompanying drawings are merely illustrative examples of the technical solution of this invention and should not be considered as the entirety of the invention or as limitations or restrictions on the technical solution of this invention.

[0018] This invention relates to the field of deep learning technology, specifically to a deep learning-based big data anomaly detection system. This system uniformly collects and analyzes multi-source behavioral data, identifies access pattern types within a preset time window, and performs differentiated anomaly detection processing based on different access patterns. Its core technology lies in constructing a hierarchical detection mechanism for regular patterns, burst patterns, and local anomaly patterns, and combining this with a global deep representation model for reconstruction error analysis. Simultaneously, it performs differentiated parameter adjustments and detection optimization processing when detecting anomalies, thereby achieving accurate identification of abnormal behavior in a big data environment.

[0019] In today's information environment, various business systems, server devices, and network platforms continuously generate massive amounts of access logs and behavioral data. This data typically exhibits characteristics such as high dimensionality, dynamic changes, and sparse distribution. Traditional anomaly detection methods often rely on fixed thresholds or single detection models. When faced with sudden access behaviors or localized anomalies, they are prone to decreased detection accuracy and increased false positive and false negative rates, making it difficult to meet the demands for real-time identification of abnormal behavior in complex big data environments. Therefore, an anomaly detection system capable of differentiated detection based on different access behavior patterns and possessing adaptive optimization capabilities is needed.

[0020] To address the aforementioned issues, this invention proposes a deep learning-based big data anomaly detection system. The system collects multi-source behavioral data from various users through a data acquisition and analysis module, aggregating and statistically analyzing the data according to a preset time window to obtain an access behavior dataset. Simultaneously, it calculates the deviation magnitude and rate of change of each behavioral data point to identify the access pattern type within the current time window. When the access pattern is non-scalar, a non-scalar detection module further analyzes the behavioral characteristics. Based on indicators such as CPU load change rate, user access traffic change rate, user request success rate, and number of error events, the non-scalar pattern is further subdivided into burst mode or local anomaly mode. Subsequently, a pattern differentiation processing module performs corresponding data anomaly detection processing. In the normal mode, a global deep representation model is used for global reconstruction error analysis. In the burst mode, an incrementally increasing sub-window is used to determine the burst starting point and construct a burst analysis time interval for reconstruction detection. In the local anomaly mode, behavioral features with high feature contribution are selected to construct a local feature subspace, and the local reconstruction error is calculated, thus obtaining a comprehensive anomaly detection result. When the overall anomaly detection result is abnormal, the anomaly detection adjustment module performs differentiated parameter adjustments based on different detection results. For example, it may shorten the time window length and adjust the detection refresh frequency, extend the burst analysis time interval, or adjust the encoding compression dimension. The anomaly detection process is then re-executed under the optimized parameter conditions to obtain optimized detection results. Finally, the anomaly warning module determines whether to trigger the corresponding warning signal based on the optimized detection results, achieving timely warnings of genuine abnormal behavior.

[0021] Through the above technical solutions, this invention can achieve differentiated anomaly detection for different access behavior patterns in complex big data environments, and continuously improve detection accuracy through an adaptive optimization mechanism for detection parameters. This effectively reduces the false alarm rate and false negative rate caused by the dilution of anomaly features in high-dimensional sparse data environments, improves the reliability and stability of the anomaly detection system, and has important application value for network security monitoring, business system operation monitoring, and user behavior analysis.

[0022] This solution includes a data acquisition and analysis module, used to collect multi-source behavioral data from each accessing user and analyze it to obtain the access mode type of the current time window, including regular mode and non-regular mode; an extraordinary detection module, used to perform behavioral feature analysis if the current time window is extraordinary, filtering out sudden mode and local anomaly mode, otherwise behavioral feature analysis is not performed; a mode differentiation processing module, used to perform corresponding data anomaly detection processing on regular mode, sudden mode, and local anomaly mode respectively, to obtain the corresponding data anomaly detection processing results, and to obtain a comprehensive anomaly detection result, including regular anomaly detection results, sudden anomaly detection results, and local anomaly detection results, and the comprehensive anomaly detection result includes detection anomaly and detection normal; an anomaly detection adjustment module, used to perform differentiated parameter adjustment and detection optimization processing based on the data anomaly detection processing results if the comprehensive anomaly detection result is detection anomaly, and to re-execute anomaly detection based on the optimized parameters to obtain optimized detection results, otherwise no processing is performed; an anomaly early warning module, used to confirm that the current time window is a real anomaly state and trigger the corresponding early warning signal when the optimized detection result is unqualified, otherwise no early warning is triggered.

[0023] In this embodiment, as Figure 1 As shown, Figure 1 The system architecture diagram of this invention includes a data acquisition and analysis module, an unscaled detection module, a pattern differentiation processing module, an anomaly detection and adjustment module, and an anomaly early warning module.

[0024] In a big data environment, various information systems continuously generate massive amounts of operational data, such as system logs, user access records, device operation monitoring data, and network request data. This data is typically characterized by its large scale, high dimensionality, rapid change, and complex sources. In actual analysis, it often involves dozens or even hundreds of behavioral characteristics, and the data distribution may vary significantly across different time periods. For example, university information systems or enterprise business systems generate a large amount of access behavior data and system operation data, including user access requests, API call records, operation logs, and system resource monitoring data. This data is typically characterized by a large number of users accessing the system, high access frequency, and complex behavioral patterns, forming large-scale high-dimensional datasets during accumulation. Under normal system operation, most access behaviors exhibit a stable distribution, but some behavioral characteristics change significantly when abnormal access, abnormal system operations, or sudden surges in business traffic occur. Therefore, it is necessary to model and analyze high-dimensional access behavior data to achieve automatic identification and early warning of abnormal behavior.

[0025] like Figure 2 As shown, Figure 2 The diagram illustrates the overall process of this invention. First, multi-source behavioral data from each user is collected and aggregated according to a preset time window to form an access behavior dataset. Then, the deviation magnitude and rate of change of the access behavior dataset are analyzed to determine whether the current time window meets the conditions for a conventional mode. If the conditions are met, the access behavior data is input into a global deep representation model for processing to obtain a global reconstruction vector. The global reconstruction error is calculated by comparing the vector with the initial access behavior feature vector, thus obtaining the conventional anomaly detection result. If the conditions are not met, behavioral feature analysis is performed, and a sudden mode determination is made. When a sudden mode is determined, the current time window is divided into multiple sub-windows with increasing increments. The sudden starting point is determined through backtracking analysis, and a sudden analysis time interval is constructed. The access behavior data within the sudden analysis time interval is globally reconstructed, and the sudden interval reconstruction error is calculated, thus obtaining the sudden anomaly detection result. When the sudden mode determination result is negative, behavioral features with high feature contribution to the reconstruction error are selected to construct a local feature subspace. Data reconstruction is performed on the local feature subspace, and the local reconstruction error is calculated, thus obtaining the local anomaly detection result. The results of routine anomaly detection, sudden anomaly detection, and local anomaly detection are then statistically analyzed to obtain a comprehensive anomaly detection result. When the comprehensive anomaly detection result is anomaly, differentiated parameter adjustment and detection optimization are performed, and the anomaly detection is re-executed after parameter adjustment to obtain an optimized detection result. If the optimized detection result is still anomaly, an early warning signal is triggered; otherwise, the process ends.

[0026] The access mode type for the current time window is determined as follows: Multi-source behavioral data is aggregated and statistically analyzed according to a preset time window to obtain an access behavior dataset. This multi-source behavioral data includes each user's dwell time, operation interval duration, API call frequency, error event frequency, and total number of access requests. A mean is processed based on the access behavior dataset to obtain the mean values ​​of each access behavior, representing the overall user access behavior. These mean values ​​include the mean dwell time, mean operation interval duration, mean API call frequency, mean error event frequency, and mean total number of access requests. Continuous time window parsing is performed based on the access behavior dataset to obtain the deviation magnitude and rate of change of each multi-source behavioral data point. If the deviation magnitude of each multi-source behavioral data point is below its corresponding deviation magnitude threshold and the rate of change is below its corresponding rate of change threshold, then the access mode type for the current time window is the normal mode; otherwise, it is the non-normal mode.

[0027] In this embodiment, it should be noted that the mean processing based on the access behavior dataset specifically involves performing mean processing on each of the multi-source behavior data within the access behavior dataset. For example, the mean of each user's dwell time is processed to obtain the mean dwell time, and the mean of each user's operation interval time is processed to obtain the mean operation interval time, etc.

[0028] User dwell time refers to the duration of a single user's session from entering the system to leaving the system. Operation interval duration refers to the time difference between two consecutive valid operations by the same user. Interface access call frequency refers to the number of times a system interface is called per unit of time. Error event frequency refers to the number of times system access is rejected or fails within a preset time period. Each user's dwell time, operation interval duration, interface access call frequency, error event frequency, and total number of access requests can all be obtained by analyzing the access logs of all users within a preset time window in data management.

[0029] The duration of each user's stay, the duration of operation intervals, the frequency of API calls, the frequency of error events, and the total number of access requests collectively constitute the correlation structure between access intensity, access rhythm, and system stability. The total number of access requests is positively correlated with the frequency of API calls; an increase in the total number of access requests directly drives up the frequency of API calls. When the frequency of API calls continues to rise, if the system's processing capacity is sufficient, the frequency of error events remains stable, typically indicating busy but normal operation. If the frequency of API calls rises simultaneously with the frequency of error events, it indicates that the system resources are approaching saturation, entering an abnormal or overloaded state. The duration of operation intervals is negatively correlated with the total number of access requests; the more frequent the requests, the shorter the operation interval, reflecting a faster user access rhythm. If the operation interval shortens significantly but the duration of stay does not increase significantly, it usually indicates short-term, high-frequency access behavior. If the duration of stay increases significantly and the operation interval shortens, it indicates increased user engagement or concentrated processing behavior. When access intensity increases while the frequency of error events remains stable, it is mostly a normal business peak. When access intensity does not change much but the frequency of error events increases abnormally, it indicates a local system anomaly or functional failure.

[0030] Mean processing eliminates random fluctuations caused by individual user differences, elevating the analysis from individual behavior to overall access behavior and enhancing judgment stability. Secondly, deviation magnitude reflects long-term stability, while the rate of change reflects short-term trend changes, enabling joint judgment of static benchmarks and dynamic trends, thus improving the accuracy of pattern recognition. By completing access pattern screening before entering deep learning anomaly detection, false triggering of anomaly detection during peak business conditions can be avoided, thereby reducing false alarm rates and unnecessary model computation overhead. Finally, a hierarchical detection structure is achieved.

[0031] Furthermore, burst patterns and local anomaly patterns are identified through screening. Specifically, the instantaneous resource dataset for the current time window is obtained, which includes CPU load rate, user access traffic, user request success rate, and number of error events. The CPU load change rate and user access traffic change rate are then analyzed. If the CPU load change rate and / or user access traffic change rate exceed the corresponding change rate threshold, and the user request success rate is above the user request success rate threshold and the number of error events is less than the error event number threshold, then the anomaly pattern is determined to be a burst pattern; otherwise, it is determined to be a local anomaly pattern.

[0032] In this embodiment, CPU load rate is collected in real time by the background server monitoring system, originating from the operating system performance monitoring interface, and the average load value for the current time period is directly output by the CPU utilization statistics module. User access traffic is obtained through the background network traffic monitoring module, which can statistically analyze the amount of data transmitted or the number of request packets per unit time based on the gateway or load balancing device. User request success rate is obtained through background application log statistics, calculated by dividing the number of successfully responded requests per unit time by the total number of requests. The number of error events is obtained through system error log statistics, recording the number of error records generated within a unit time window.

[0033] Within the current time window, CPU load rate and user access traffic data are collected from the instantaneous resource dataset and a continuous time series is established. Both the CPU load change rate and the user access traffic change rate are calculated based on data from adjacent time windows. Specifically, the CPU load change rate is obtained by subtracting the CPU load rate of the previous time window from the CPU load rate of the current time window and then dividing by the CPU load rate of the previous time window. Similarly, the user access traffic change rate is obtained by subtracting the user access traffic of the previous time window from the user access traffic of the current time window and then dividing by the user access traffic of the previous time window.

[0034] When the CPU load change rate and / or user access traffic change rate increase significantly, it indicates a rapid increase in overall system resource consumption or access volume. If the user request success rate remains above a preset threshold and the number of error events is low, it suggests that the system is operating normally despite significant pressure. This is attributed to a surge in business volume or a sudden increase in external traffic, thus classifying it as a burst mode. Conversely, if resource changes are not significant, or if changes occur but are accompanied by a decrease in the user request success rate or an increase in the number of error events, it indicates an abnormal performance at the system functional level. This is a structural fault or a localized anomaly, rather than a simple increase in traffic, and is therefore classified as a localized anomaly mode.

[0035] This solution further subdivides anomaly detection into burst mode and local anomaly mode after identifying it as an extraordinary event, enabling refined traffic routing in the anomaly detection process. Identifying sudden increases in overall system load through resource change rate avoids misjudging peak business periods as system anomalies. Introducing user request success rate and error event count as functional stability constraints accurately identifies abnormal system states, improving judgment accuracy. Furthermore, using CPU load rate and user access traffic as the basis for division is because these directly reflect changes in system resource consumption intensity and access scale, providing a better early warning of system operating trends compared to simply using outcome indicators such as error count. This mode subdivision mechanism allows subsequent anomaly detection to adopt differentiated processing strategies based on different scenarios, improving overall detection efficiency and false alarm control capabilities, and enhancing the stability and adaptability of the anomaly detection system in big data environments.

[0036] Furthermore, the comprehensive anomaly detection results are obtained statistically. The specific method is as follows: Based on access mode type analysis, if the access mode type is normal, a global deep representation model is used to analyze and obtain normal anomaly detection results, including normal detection pass and normal detection fail. If the access mode type is burst mode, a time window adjustment process is performed to obtain burst anomaly detection results, including burst detection pass and burst detection fail. If the access mode type is local anomaly mode, a local feature subspace modeling process is performed to obtain local anomaly detection results, including local detection pass and local detection fail. The combined results of normal anomaly detection, burst anomaly detection, and local anomaly detection are collectively recorded as the comprehensive anomaly detection result. If the normal anomaly detection result is normal detection pass, the burst anomaly detection result is burst detection pass, or the local anomaly detection result is local detection pass, then the corresponding comprehensive anomaly detection result is anomaly detection pass; otherwise, the comprehensive anomaly detection result is anomaly detection fail.

[0037] In this embodiment, the global deep representation model performs overall representation and reconstruction analysis of multi-source behavioral data within a unified feature space to determine whether the access behavior in the current time window deviates from the normal operating structure of the system. In this scheme, when the access mode is determined to be a normal mode, a global reconstruction analysis is directly performed on the access behavior dataset, outputting normal anomaly detection results. In burst mode, after time window adjustment, a global reconstruction analysis is performed on the reconstructed burst analysis time interval data to determine whether burst growth is accompanied by structural anomalies. Furthermore, in the subsequent optimization stage of local anomaly modes, this global deep representation model serves as the basic coding framework for adjusting the coding compression dimension, providing structural support for the reconstruction of local feature subspaces.

[0038] The global deep representation model employs an encoder-reconstruction structure. Its input is an initial access behavior feature vector formed from the access behavior dataset within the current time window. This feature vector includes normalized average dwell time, average operation interval duration, average interface access call frequency, average error event frequency, and average total number of access requests. The model structure comprises an input layer, an encoder layer, a latent representation layer, and a decoder layer. The encoder layer compresses the input features using multiple nonlinear mapping operators. The latent representation layer forms a low-dimensional global deep representation vector. The decoder layer uses a symmetric mapping structure to restore the low-dimensional representation to a global reconstruction vector of the same dimension as the input. During model training, historical normal time window data is used as training samples, and parameter optimization is performed with the objective function of minimizing the reconstruction error. During model execution, the current time window feature vector is input into the model, and the corresponding global reconstruction vector is output. Anomaly detection is achieved by calculating the global reconstruction error between the initial access behavior feature vector and the global reconstruction vector. This structure ensures that the model can learn the inherent correlation patterns of normal access behavior. When the input data deviates from the existing structural distribution, the reconstruction error will increase significantly, thus enabling the identification of overall abnormal states.

[0039] In normal mode, the system operates stably, making a global deep representation model suitable for overall reconstruction analysis. This is because the data distribution is relatively stable, and the reconstruction error effectively reflects the true anomaly. In burst mode, the overall system access intensity changes rapidly within a short period. Using a fixed time window for global reconstruction directly might misjudge peak traffic as anomalies. Therefore, adjusting the time window to align the analysis interval with the burst's starting point helps identify whether the actual burst growth is accompanied by structural anomalies. In local anomaly mode, anomalies are usually concentrated in certain behavioral feature dimensions, while the overall behavioral structure may not have changed drastically. Therefore, using a local feature subspace modeling method for targeted reconstruction analysis of high-contribution features improves the ability to identify hidden local anomalies. Global modeling for the overall stable state, time interval reconstruction analysis for the overall fluctuating state, and refined feature subspace modeling for local anomaly states can reduce false alarms while ensuring detection accuracy and improving system resource utilization efficiency.

[0040] The essential characteristic of burst patterns is a rapid increase in overall access volume or system load within a short period, while system functionality may remain normal. Therefore, the key to anomaly detection lies in distinguishing between traffic growth and structural anomalies. Adjusting the time window allows for pinpointing the burst's origin and reconstructing the analysis interval, enabling the detection model to reconstruct and analyze based on the actual change segment, thus avoiding misjudgments due to inappropriate time interval selection. Directly modeling local feature subspaces during burst patterns can amplify local errors due to changes in feature proportions caused by overall traffic fluctuations, ultimately reducing accuracy.

[0041] Local anomaly models are characterized by abnormal behavior in some metrics, while the overall access volume remains relatively stable. In such cases, simply adjusting the time window cannot effectively amplify the differences in anomaly features. However, by constructing a local feature subspace through selecting behavioral features with higher contribution, the focus can be placed on the anomaly dimension, preventing the anomaly signal from being diluted during global modeling. Therefore, distinguishing between anomaly structural differences can effectively prevent sudden traffic spikes from being misjudged as system anomalies, and also prevent local anomalies from being misjudged as normal traffic fluctuations, thereby improving the accuracy and stability of anomaly detection.

[0042] Furthermore, the results of routine anomaly detection are obtained by: inputting the access behavior dataset into a pre-stored global deep representation model for processing to obtain a global reconstruction vector; retrieving the initial access behavior feature vector and comparing it with the global reconstruction vector to obtain the global reconstruction error; when the global reconstruction error is above the preset reconstruction error threshold, the routine anomaly detection result is deemed unqualified, otherwise it is deemed qualified.

[0043] In this embodiment, the global reconstruction vector is obtained through the following process: the access behavior dataset formed within the current time window is processed by feature vectorization and normalization, then input into the input layer of a pre-stored global deep representation model. The dataset is compressed and mapped to the latent representation space by the encoding layer, and then restored to a reconstruction output with the same dimension as the input by the decoding layer, thus obtaining the global reconstruction vector. Its effect is to characterize the difference between the current access behavior and the historical normal behavior structure through the reconstruction mechanism, providing a unified structural benchmark for subsequent anomaly detection.

[0044] The initial access behavior feature vector originates from the access behavior dataset generated by the data acquisition and analysis module within the current time window. The specific steps are as follows: After time window aggregation and averaging, each behavior parameter (average dwell time, average operation interval duration, average interface access call frequency, average error event frequency, and average total number of requests) is combined in a fixed feature order to form a structured feature vector, which is then stored in the feature cache unit of the current detection process. When the global deep representation model outputs the global reconstruction vector, the initial access behavior feature vector for the corresponding time window is retrieved from this feature cache unit and compared dimension-by-dimensionally with the reconstruction vector. The global reconstruction error is calculated using the mean squared error method, i.e., the squared difference of each feature dimension is calculated and averaged to obtain the overall error value. This error is used to quantify the degree of deviation between the current behavior and the normal structure learned by the model.

[0045] The global deep representation model optimizes its parameters based on historical normal data during the training phase. Its goal is to minimize the reconstruction error of normal behavioral data. When the behavioral features of the current time window conform to the historical normal distribution, the model can accurately reconstruct the behavior, keeping the reconstruction error at a low level. However, when the behavioral structure deviates abnormally, the model cannot effectively reconstruct the feature structure, leading to a significant increase in reconstruction error. Therefore, a reconstruction error threshold is set as the judgment boundary. When the error exceeds the threshold, it indicates that the current behavior is outside the normal distribution range, and it is judged as failing the routine detection. When the error is below the threshold, it indicates that the behavior is still within the reconstructable range, and it is judged as passing the routine detection.

[0046] By generating a global reconstruction vector and calculating the global reconstruction error, a unified quantitative evaluation of the overall access behavior structure is achieved. The purpose of adopting the reconstruction mechanism is to learn the potential feature correlations of normal behavior distribution through the model, rather than relying on a single indicator for threshold judgment, thereby improving the overall accuracy and stability of anomaly identification. Judging pass / fail based on the reconstruction error can effectively identify anomalies in the overall behavior structure, avoid misjudgments caused by fluctuations in a single parameter, and provide a clear source of anomaly signals for subsequent pattern differentiation processing.

[0047] Furthermore, the sudden anomaly detection results are obtained through the following method: the current time window is divided into multiple sub-windows with increasing increments according to preset rules; the mean and rate of change of the access behavior dataset within each sub-window are calculated; starting from the most recent time point, the process is repeated step by step, and when the difference in the rate of change between two and / or more consecutive adjacent sub-windows exceeds a preset progressive change threshold, the earliest sub-window that meets the condition is locked as the sudden anomaly starting point; the sudden analysis time interval is reconstructed based on the sudden anomaly starting point, where the end point of the sudden analysis time interval is the end point of the current time window; the access behavior dataset within the sudden analysis time interval is input into the global deep representation model for global reconstruction to obtain the sudden interval reconstruction error. If the sudden interval reconstruction error exceeds a preset threshold, the sudden anomaly detection result is considered unqualified; otherwise, the sudden anomaly detection is considered qualified.

[0048] In this embodiment, as Figure 3 As shown, Figure 3The flowchart for the encoding compression dimension adjustment and local anomaly optimization detection of this invention is as follows: The initial access behavior feature vector corresponding to the current time window is obtained, and the global reconstruction vector output by the global deep representation model is also obtained. The global reconstruction error is calculated by comparing the initial access behavior feature vector and the global reconstruction vector, and the global reconstruction error is decomposed into components to obtain the feature contribution of each access behavior feature. Feature sparsity evaluation is performed on the initial access behavior feature vector, and anomaly candidate features are screened based on feature contribution and feature sparsity to increase the weight of anomaly candidate features in the encoding layer. The error deviation rate between the global reconstruction error and the local reconstruction error is calculated to obtain the optimized global deep representation model. Based on the optimized global deep representation model, the local feature subspace reconstruction is re-executed, and the updated local reconstruction error is calculated. The updated local reconstruction error is output as the optimized detection result, and the process ends.

[0049] The sudden mode is characterized by a sudden change in behavior but the system function remains normal. If the sudden start point is not accurately identified, "gradual growth" may be mistaken for "sudden anomaly". Therefore, by tracing back step by step to lock the sudden start point, the noise of the gradual growth stage can be removed and the influence of statistical error can be eliminated, and a more accurate analysis interval can be reconstructed.

[0050] A successful burst test indicates that the structural changes during that period were caused by variations in business volume, but the system did not exhibit any abnormal behavior. This is not due to a localized anomaly, but rather a normal fluctuation in access.

[0051] The default rule is to construct a nested sub-window structure with increasing length based on a fixed minimum time granularity. Specifically, the total length of the current time window is set as follows: Set the minimum analysis granularity Using the current time window's end point as the alignment reference, multiple sub-windows are constructed sequentially forward from the end point, with the sub-window lengths as follows: Until the entire time window is covered Each sub-window has the same endpoint, which is the endpoint of the current time window. Only the starting point is progressively moved forward, thus forming a time segment sequence of increasing length. This structure ensures that all sub-windows focus on the most recent behavior, while enabling retrospective analysis of the formation process of sudden changes by progressively expanding the observation interval.

[0052] The mean and rate of change of the access behavior dataset within each sub-window are calculated separately. Specifically, for each incrementally growing sub-window, the access behavior dataset within that sub-window is extracted, and the mean of each behavior parameter is calculated by summing all sample values ​​of that parameter within the window and dividing by the number of samples to obtain the sub-window mean vector. Then, the rate of change between adjacent sub-windows is calculated. When the difference in the rate of change between two or more consecutive adjacent sub-windows exceeds a preset gradual change threshold, it indicates that the behavior growth exhibits a non-smooth transition, consistent with the characteristics of sudden growth.

[0053] By using incrementally increasing sub-windows to trace back and identify the earliest significant change in the sub-window, and then using this starting point as the burst analysis time interval to reconstruct the burst starting point, the true start time of the burst behavior can be located more quickly and accurately. If the entire original time window is used directly for modeling, the burst features in the later period will be diluted by the stable data in the earlier period, reducing the detection sensitivity. However, by tracing back step by step through incrementally increasing sub-windows, the process of the mutation can be traced from the "current state" backward, avoiding mistaking gradual growth for a sudden anomaly. This ensures that the detection focus is on the current behavior, and the formation path of the change trend can be identified by expanding the window, thereby improving the accuracy and temporal resolution of burst location.

[0054] The burst interval reconstruction error is obtained as follows: After identifying the burst origin, the data between the burst origin and the end of the current time window is re-aggregated to form a dataset of access behavior within the burst analysis time interval. The mean of the data within this interval is calculated according to a predetermined feature order, and a burst interval feature vector is constructed. After normalization, this vector is input into a pre-stored global depth representation model. The model outputs the corresponding burst interval reconstruction vector, which is then compared dimension-by-dimensionally with the original burst interval feature vector. The burst interval reconstruction error is calculated using the mean squared error method. If this error exceeds a preset threshold, the burst detection is deemed unqualified; otherwise, the burst detection is deemed qualified.

[0055] By combining an incremental sub-window backtracking mechanism with reconstruction analysis, refined identification of sudden behaviors is achieved. The location of abrupt changes is identified through the difference in the rate of change, avoiding positioning errors caused by simple time truncation. Secondly, by constructing a dedicated sudden analysis time interval, the proportion of sudden features in the model input is increased, improving the sensitivity of reconstruction error judgment. A global deep representation model is used to perform structural verification of the sudden interval, distinguishing between "normal business surges" and "abnormal structural changes." This effectively reduces the risk of false alarms caused by business peaks, while improving the ability to identify real abnormal sudden events, enhancing the stability and judgment accuracy of the anomaly detection system in highly dynamic environments.

[0056] Furthermore, the local anomaly detection results are obtained. The specific method is as follows: behavioral features whose feature contribution to the reconstruction error is higher than the preset contribution threshold are selected to form a local feature subspace; the local feature subspace data is reconstructed, and the local reconstruction error of the local feature subspace is analyzed; when the local reconstruction error exceeds the preset local anomaly threshold, the local anomaly detection result is considered unqualified, otherwise it is considered qualified.

[0057] In this embodiment, the local feature subspace is constructed as follows: Within the current time window, an access behavior dataset is acquired, and an initial access behavior feature vector is constructed according to a predetermined feature order. This feature vector is then input into a global deep representation model to obtain the corresponding global reconstruction vector. The single-feature reconstruction error is calculated dimension-by-dimensionally for both the original and reconstructed feature vectors. Specifically, the calculation method is the squared difference between the original and reconstructed values ​​of each feature. The single-feature reconstruction errors of all features are summed to obtain the overall reconstruction error, and the feature contribution of each feature is calculated. The feature contribution is defined as the proportion of the single-feature reconstruction error to the overall reconstruction error. The contribution of each feature is compared with a preset contribution threshold. When the contribution of a certain behavior feature exceeds this threshold, the behavior feature is marked as an anomalous reconstruction feature. All marked anomalous reconstruction features are combined according to their original order to construct a local feature subspace vector set, thereby forming a feature subspace specifically for local anomaly analysis.

[0058] The local feature subspace data is reconstructed, and the local reconstruction error is analyzed. Specifically, for the constructed local feature subspace data, the original sample data within the corresponding time window is extracted, and each feature is normalized before being input into the local reconstruction model for feature reconstruction. The local reconstruction model employs a simplified autoencoder network. Its encoding layer maps the local feature subspace to a low-dimensional representation space, and the decoding layer performs an inverse mapping on the low-dimensional representation to generate a local reconstruction vector. After the model outputs the local reconstruction vector, it is compared dimension-by-dimensionally with the corresponding original local feature subspace vector. The local reconstruction error is obtained by calculating the squared error and taking the mean square value. This local reconstruction error is then compared with a preset local anomaly threshold. When the local reconstruction error exceeds the threshold, the local anomaly detection result is deemed unqualified; if it does not exceed the threshold, the local anomaly detection result is deemed qualified.

[0059] Analyzing local anomaly detection results allows for independent judgment of a small number of key feature anomalies. When anomalies are concentrated only in certain behavioral dimensions, the reconstruction results of the global model may be diluted by a large number of normal features, resulting in the overall reconstruction error remaining within the normal range. However, the local feature subspace reconstruction mechanism focuses on anomalous features with higher contribution, giving these anomalous features higher weight in the analysis process, making them easier to identify. Therefore, when the local reconstruction error increases significantly, it can be determined that there is real anomalous behavior, thus preventing fine-grained anomalies from being ignored by the global analysis and improving the sensitivity of anomaly detection.

[0060] By constructing a local feature subspace and performing local reconstruction analysis, the problem of anomaly dilution in high-dimensional feature environments can be effectively solved. When the dimension of access behavior data is high, a small number of anomalous features are easily masked by a large number of normal features in the overall statistics, leading to a decrease in global detection accuracy. By extracting anomaly-sensitive features through a feature contribution screening mechanism and independently modeling them in the local feature subspace, the detection process is transformed from "global average judgment" to "key feature focused analysis." This complements the conventional mode and the burst mode: the conventional mode focuses on judging the overall behavioral structure, the burst mode focuses on detecting abrupt changes in time, and the local anomaly mode specifically identifies local dimensional anomalies, thus forming a multi-level anomaly detection system and improving the overall detection accuracy and stability of the system.

[0061] Furthermore, based on the data anomaly detection processing results, differentiated parameter adjustments and detection optimization processing are performed. Specifically, if the data anomaly detection processing result is a regular detection failure, the current time window length is shortened based on the first preset window adjustment step size, and the detection refresh is adjusted according to the preset refresh adjustment frequency; otherwise, the time window adjustment is not performed. If the data anomaly detection processing result is a sudden detection failure, the sudden analysis time interval is extended. If the data anomaly detection processing result is a local detection failure, the encoding compression dimension is adjusted. Based on the optimized parameters, the corresponding mode of anomaly detection processing is re-executed to obtain optimized detection results.

[0062] In this embodiment, the burst analysis time interval is extended by means of: taking the burst start point of the current burst analysis time interval as the reference point and extending it forward by one sub-window length in the historical time direction. If no earlier burst start point is generated after the new sub-window is added, the extension continues to be extended backward by one sub-window length until the preset maximum number of backtracking sub-windows is reached. The extended burst analysis time interval is taken from the finally determined burst start point to the end point of the current time window. The sub-window length is consistent with the sub-window division rule when locking the burst start point.

[0063] When the data anomaly detection result indicates a failure in routine detection, the system gradually shortens the current time window length based on a first preset window adjustment step size. For example, it progressively reduces the analysis time range according to a fixed ratio, while simultaneously increasing the anomaly detection execution frequency according to a preset refresh adjustment frequency. This allows the system to update and analyze access behavior data within shorter time intervals. The time window shortening ends when the global reconstruction error falls below the reconstruction error threshold for multiple consecutive detection cycles, or when the current time window length reaches the minimum window length set by the system. The detection refresh frequency adjustment ends when consecutive detection results return to a normal detection pass state, or when the detection refresh frequency reaches the maximum refresh frequency set by the system. By simultaneously adjusting the time window length and the detection refresh frequency, the temporal resolution of monitoring can be improved when the system exhibits continuous abnormal behavior, enabling faster identification of abnormal changes. This avoids the computational resource consumption caused by maintaining high-frequency detection for extended periods, thus achieving a balance between detection speed and system resources.

[0064] When the data anomaly detection process fails to detect a sudden surge, extending the surge analysis time interval allows the anomaly detection model to incorporate access behavior data over a longer period for comprehensive analysis. This helps determine whether the current traffic surge is caused by persistent abnormal behavior. For example, when access traffic spikes dramatically within a short period, analysis based solely on a short time window may struggle to distinguish between genuine abnormal behavior and short-term business peaks. Extending the surge analysis time interval allows observation of the sustained changes in surge behavior over a longer timeframe, thereby improving the accuracy of sudden anomaly identification.

[0065] When the data anomaly detection processing result is that the local detection is unqualified, the encoding compression dimension adjustment is performed. By reducing the compression ratio of the encoding layer or increasing the representation dimension of key features, the fine-grained anomaly features that were originally compressed and hidden can be more fully expressed in the feature representation space, thereby reducing the masking of anomaly information during high-dimensional data compression and improving the accuracy of local anomaly identification.

[0066] Differentiated parameter adjustments are implemented for different types of anomaly detection results because the mechanisms underlying the three types of anomalies differ significantly. Routine detection failures typically indicate persistent abnormal changes in overall system access behavior. In this case, shortening the time window and increasing the detection refresh frequency improves the detection response speed, enabling the system to quickly capture the abnormal trend. Sudden detection failures are usually caused by large-scale access fluctuations within a short period. Extending the sudden analysis time interval expands the observation scope, allowing for a determination of whether the sudden behavior exhibits persistent abnormal characteristics. Localized detection failures usually stem from abnormal changes in a few key behavioral features. Adjusting the encoding compression dimension enhances the expressive power of key features in the feature representation space, making localized anomalies easier to detect. By adopting different parameter adjustment strategies for different anomaly patterns, the anomaly detection process becomes more accurate and efficient, thereby improving the stability and reliability of the entire big data anomaly detection system.

[0067] Furthermore, the encoding compression dimension is adjusted, specifically as follows: The initial access behavior feature vector and its global reconstruction vector corresponding to the current time window are obtained; the global reconstruction error is calculated based on the initial access behavior feature vector and the global reconstruction vector, and the global reconstruction error is decomposed into components to obtain the feature contribution of each access behavior feature, where the feature contribution is the proportion of the single feature reconstruction error to the overall reconstruction error; the sparsity of the initial access behavior feature vector is evaluated to obtain the sparsity of each access behavior feature; access behavior features with feature contribution exceeding a contribution threshold and feature sparsity exceeding a sparsity threshold are selected as anomaly candidate features; a weighted coefficient matrix is ​​constructed based on the feature contribution and feature sparsity of the anomaly candidate features, and an attention mechanism is used to adaptively weight the input features of the encoding layer to increase the weight coefficients of the anomaly candidate features in the encoding layer of the global deep representation model; the compression ratio of the encoding layer is reduced according to the error deviation rate between the global reconstruction error and the local reconstruction error to obtain the optimized global deep representation model; the local feature subspace reconstruction is re-executed based on the optimized global deep representation model, and the updated local reconstruction error is calculated; the updated local reconstruction error is used as the optimized detection result.

[0068] In this embodiment, the feature contribution of each access behavior feature is obtained as follows: Within the current time window, user access behavior data is constructed into an initial access behavior feature vector, and encoded-decoded using a global deep representation model to obtain the corresponding global reconstruction vector. The single-dimensional reconstruction error for each feature dimension is calculated, i.e., the squared difference between the original value and the reconstructed value for each access behavior feature is calculated to obtain the single-dimensional error component of each feature. The error components of all dimensions are summed to obtain the overall global reconstruction error. Finally, the single-dimensional error component of each feature is proportionalized to the overall reconstruction error to obtain the proportion of a single access behavior feature to the overall reconstruction error. This proportion is the feature contribution of the corresponding access behavior feature, thereby decomposing the global reconstruction error into components and quantifying the influence of each feature on abnormal reconstruction deviations.

[0069] The sparsity of each access behavior feature is obtained as follows: After obtaining the initial access behavior feature vector, the value distribution of each access behavior feature within the current time window is statistically analyzed. The number of non-zero values ​​and the total number of samples for each feature in the window time series are counted, and the proportion of non-zero values ​​is calculated. Combined with the variance of the feature, the effective information density of the feature is evaluated. When an access behavior feature is at or near zero (the difference from zero is less than a preset difference) for most of the time within the time window (most of the time, i.e., the cumulative time exceeds the preset time length), the feature is determined to have high sparsity. By comparing and dividing the non-zero proportion with a preset sparsity threshold, the sparsity index of each access behavior feature is obtained, thereby identifying feature dimensions that exhibit sparse distribution in the time series but have potential anomaly indication capabilities.

[0070] To enhance the weight coefficients of anomalous candidate features in the encoding layer of the global deep representation model, the specific process is as follows: After selecting anomalous candidate features that meet the feature contribution and sparsity thresholds, the contribution and sparsity of these features are first normalized, and corresponding feature weight coefficients are generated through weighted fusion. This constructs a weighted coefficient matrix, which is then input to the pre-stored attention weight calculation module at the front end of the encoding layer. The attention mechanism assigns weights to the input feature vector, increasing the activation intensity of anomalous candidate features in feature encoding calculation, while maintaining the basic weights of ordinary features. Specifically, at the input of the encoding layer, the original feature vector is multiplied dimension-by-dimensionally by the weight coefficients to form a weighted input feature vector. This weighted feature vector is then input into the encoding network for feature compression representation, thereby achieving a focused and enhanced representation of anomalous candidate features.

[0071] Based on the error deviation rate between the global reconstruction error and the local reconstruction error, the compression ratio of the coding layer is reduced to obtain an optimized global deep representation model. Specifically, the global reconstruction error and the local feature subspace reconstruction error are calculated separately, and an error deviation rate index is constructed by comparing the difference between the two with the global reconstruction error. This index measures the degree of reconstruction difference between the overall feature space and the local feature subspace. When the error deviation rate exceeds a preset threshold, it indicates that the current compression ratio of the coding layer is too high, causing excessive compression of local anomalous features. In this case, the compression ratio of the coding layer is gradually reduced by a preset compression adjustment ratio, i.e., the feature dimensionality reduction of the coding layer is reduced, allowing the coding network to retain more original feature information. This results in an optimized global deep representation model that better preserves local anomalous feature information while maintaining overall representational capability.

[0072] The local feature subspace reconstruction is re-executed, and the updated local reconstruction error is calculated. Specifically, after obtaining the optimized global deep representation model, the feature dimensions corresponding to the anomaly candidate features are constructed as a local feature subspace, and the data vectors of these features within the current time window are extracted. This local feature subspace data is then input into the optimized model's encoding layer for feature compression, and the compressed feature representation is reconstructed in reverse through the decoding layer to obtain the corresponding local reconstruction vector. The original feature vector of the local feature subspace and its corresponding reconstruction vector are then subtracted dimension by dimension, and the differences are squared and summed. Finally, the average of the errors across all dimensions is calculated to obtain the updated local reconstruction error. The local reconstruction error is used to quantify the accuracy of the optimized model's reconstruction in the local feature subspace.

[0073] The updated local reconstruction error is a quantitative indicator used for anomaly detection. In practice, the updated local reconstruction error is first compared with a preset anomaly threshold. If the error exceeds the threshold, the access behavior within the current time window is considered abnormal; otherwise, it is considered normal access behavior. Therefore, the updated local reconstruction error, as a core indicator for optimizing detection results, reflects the degree of reconstruction deviation of anomaly candidate features under the optimized model. Combined with the threshold determination mechanism, it ultimately yields the anomaly detection conclusion.

[0074] The corresponding warning signals are triggered as follows: when the optimized detection result is unqualified, the current time window is confirmed to be a real abnormal state and a first warning signal is issued; when the access mode is a local abnormal mode and the optimized detection result is unqualified, a second warning signal is triggered; when the access mode is a burst mode and the optimized detection result is qualified, no warning is triggered.

[0075] In this embodiment, the triggering conditions for different warning signals are designed differently based on the type of abnormal access behavior and its risk characteristics. This aims to avoid false alarms or distortion of warning levels caused by a unified warning mechanism, thereby improving the accuracy and targeted response of the anomaly detection system. Specifically, when the optimized detection result is unqualified, it indicates that the reconstruction error after reconstruction using the global depth representation model and local feature subspace within the current time window has exceeded the anomaly judgment threshold. The system confirms that the overall access behavior within this time window deviates from the normal behavior pattern, thus determining the current time window as a true anomaly and triggering the first warning signal. The first warning signal corresponds to the overall abnormal access behavior risk and is typically used to alert the system management module or security monitoring module to potential system-level abnormal access behavior, such as continuous abnormal requests, abnormal access structures, or abnormal call patterns. When the access pattern is identified as a local anomaly pattern and the optimized detection result is unqualified, a second warning signal is triggered. This is because local anomaly patterns typically manifest as abrupt changes in a few access behavior feature dimensions, such as significant deviations in interface access call frequency, error event frequency, or access request structure in local dimensions, while the overall access behavior still retains some normal characteristics. Using only a unified early warning mechanism can easily lead to local anomalies being misjudged as overall anomalies. Therefore, a second early warning signal is used to separately identify local anomalies, allowing the system to focus on the behavioral dimensions corresponding to anomaly candidate features, thereby achieving more accurate anomaly localization and subsequent analysis and processing. On the other hand, when the access pattern is identified as a burst mode and the optimized detection result is qualified, no early warning is triggered. This is because burst modes usually correspond to a sudden increase in the number of access requests or the intensity of access behavior within a short period of time, such as business peaks, batch task execution, or normal traffic surges. In such cases, although the access behavior shows burst characteristics in statistical distribution, it still does not exceed the anomaly judgment threshold after global deep representation model and local reconstruction error analysis. Therefore, it can be determined that the behavior belongs to normal business fluctuations rather than abnormal access behavior. If an early warning is still triggered in this case, it will lead to a large number of false alarms, affecting the stability and reliability of the anomaly detection system. Therefore, by setting differentiated early warning mechanisms for three different access behavior states—true anomaly state, local anomaly mode, and burst mode—the anomaly detection system can effectively reduce the probability of false alarms while maintaining detection sensitivity, and improve the accuracy of anomaly behavior identification and risk response.

[0076] The technical scope of this invention is not limited to the content described above. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the protection scope of this invention.

Claims

1. A big data anomaly detection system based on deep learning, characterized in that: Including the following: The data acquisition and analysis module is used to collect multi-source behavioral data of each accessing user and analyze it to obtain the access mode type of the current time window. The access mode type includes regular mode and non-regular mode. The non-scale detection module is used to perform behavioral feature analysis if the current time window is non-scale, and to screen out burst patterns and local abnormal patterns; otherwise, behavioral feature analysis is not performed. The mode differentiation processing module is used to perform corresponding data anomaly detection processing on the normal mode, the burst mode and the local anomaly mode respectively, obtain the corresponding data anomaly detection processing results, and statistically obtain the comprehensive anomaly detection results. The data anomaly detection processing results include normal anomaly detection results, burst anomaly detection results and local anomaly detection results, and the comprehensive anomaly detection results include detected anomalies and detected normalities. The anomaly detection and adjustment module is used to perform differentiated parameter adjustment and detection optimization processing based on the data anomaly detection processing results if the comprehensive anomaly detection result is an anomaly, and then re-execute the anomaly detection based on the optimized parameters to obtain the optimized detection result; otherwise, no processing is performed. The anomaly warning module is used to confirm that the current time window is a real anomaly and trigger the corresponding warning signal when the optimized detection result is unqualified; otherwise, no warning is triggered.

2. The deep learning-based big data anomaly detection system according to claim 1, characterized in that: The specific method for obtaining the access mode type of the current time window is as follows: The multi-source behavioral data is aggregated and statistically analyzed according to a preset time window to obtain the access behavior dataset. The mean value is calculated by performing mean processing on the access behavior dataset to obtain the mean value of each access behavior that represents the overall user access behavior. Continuous time window parsing is performed based on the access behavior dataset to obtain the deviation magnitude and rate of change of each multi-source behavior data. If the deviation magnitude of each multi-source behavior data is below the corresponding deviation magnitude threshold and the rate of change is below the corresponding rate of change threshold, then the access mode type of the current time window is the normal mode; otherwise, it is the non-normal mode.

3. The deep learning-based big data anomaly detection system according to claim 1, characterized in that: The screening process yields burst patterns and local anomaly patterns, specifically through the following method: Obtain the instantaneous resource dataset for the current time window, which includes CPU load rate, user access traffic, user request success rate, and number of error events, and analyze the CPU load change rate and user access traffic change rate from there. If the CPU load change rate and / or user access traffic change rate exceed the corresponding change rate threshold, and the user request success rate is above the user request success rate threshold and the number of error events is less than the number of error events threshold, then the abnormal mode is determined to be a burst mode; otherwise, it is determined to be a local abnormal mode.

4. The big data anomaly detection system based on deep learning according to claim 1, characterized in that: The statistical analysis yields a comprehensive anomaly detection result, specifically using the following method: Based on access mode type analysis, if the access mode type is normal mode, then the normal anomaly detection result is obtained through global deep representation model analysis. The normal anomaly detection result includes normal detection qualified and normal detection unqualified. If the access mode type is burst mode, then time window adjustment processing is performed to obtain burst anomaly detection results, which include burst detection qualified and burst detection unqualified. If the access mode type is local anomaly mode, local feature subspace modeling processing is performed to obtain local anomaly detection results, which include local detection qualified and local detection unqualified. The combined results of routine anomaly detection, sudden anomaly detection, and local anomaly detection are collectively referred to as the comprehensive anomaly detection result. If the routine anomaly detection result is "routine detection qualified", the sudden anomaly detection result is "sudden detection qualified", or the local anomaly detection result is "local detection qualified", then the corresponding comprehensive anomaly detection result is "anomaly detection qualified"; otherwise, the comprehensive anomaly detection result is "anomaly detection unqualified".

5. The deep learning-based big data anomaly detection system according to claim 4, characterized in that: The specific method for obtaining the conventional anomaly detection results is as follows: The access behavior dataset is input into a pre-stored global depth representation model for processing to obtain a global reconstruction vector; The initial access behavior feature vector is retrieved and compared with the global reconstruction vector to obtain the global reconstruction error. When the global reconstruction error is above the preset reconstruction error threshold, the routine anomaly detection result is that the routine detection is unqualified; otherwise, the routine detection is qualified.

6. The deep learning-based big data anomaly detection system according to claim 4, characterized in that: The specific method for obtaining the sudden anomaly detection results is as follows: Divide the current time window into multiple sub-windows with increasing increments according to preset rules; Calculate the mean and rate of change of the access behavior dataset in each sub-window; Starting from the most recent time point, backtracking step by step, when there are two and / or more consecutive adjacent sub-windows whose rate of change difference exceeds the preset progressive change threshold, the earliest sub-window that meets the condition is locked as the sudden start point. The burst analysis time interval is reconstructed based on the burst initiation point, wherein the end point of the burst analysis time interval is the end point of the current time window; The access behavior dataset within the burst analysis time interval is input into the global deep representation model for global reconstruction to obtain the burst interval reconstruction error. If the burst interval reconstruction error exceeds the preset threshold, the burst anomaly detection result is unqualified; otherwise, the burst detection is qualified.

7. The deep learning-based big data anomaly detection system according to claim 4, characterized in that: The specific method for obtaining the local anomaly detection results is as follows: Behavioral features whose contribution to the reconstruction error is higher than a preset contribution threshold are selected to form a local feature subspace. The local feature subspace data is reconstructed, and the local reconstruction error of the local feature subspace is analyzed. When the local reconstruction error exceeds the preset local anomaly threshold, the local anomaly detection result is considered unqualified; otherwise, the local anomaly detection is considered qualified.

8. The deep learning-based big data anomaly detection system according to claim 4, characterized in that: The specific method for performing differentiated parameter adjustment and detection optimization based on the data anomaly detection processing results is as follows: If the data anomaly detection result is that the regular detection is unqualified, the current time window length will be shortened based on the first preset window adjustment step size and the detection refresh will be adjusted according to the preset refresh adjustment frequency; otherwise, the time window adjustment will not be performed. If the data anomaly detection result is a failure to pass the sudden detection, the sudden analysis time interval will be extended. If the data anomaly detection result is that a local detection fails, then the encoding compression dimension adjustment will be performed; Based on the optimized parameters, the anomaly detection process for the corresponding mode is re-executed to obtain optimized detection results.

9. The deep learning-based big data anomaly detection system according to claim 8, characterized in that: The specific method for adjusting the execution encoding compression dimension is as follows: Obtain the initial access behavior feature vector and its global reconstruction vector corresponding to the current time window; The global reconstruction error is calculated based on the initial access behavior feature vector and the global reconstruction vector. The global reconstruction error is then decomposed into components to obtain the feature contribution of each access behavior feature. The feature contribution is the proportion of the single feature reconstruction error to the overall reconstruction error. Sparsity evaluation is performed on the initial access behavior feature vector to obtain the sparsity of each access behavior feature; Access behavior features whose contribution is higher than the contribution threshold and whose sparsity is higher than the sparsity threshold are selected as abnormal candidate features. A weighted coefficient matrix is ​​constructed based on the feature contribution and feature sparsity of the abnormal candidate features. An attention mechanism is used to adaptively weight the input features of the encoding layer, thereby increasing the weight coefficient of the abnormal candidate features in the encoding layer of the global deep representation model. Based on the error deviation rate between the global reconstruction error and the local reconstruction error, the compression ratio of the coding layer is reduced to obtain the optimized global depth representation model. Based on the optimized global depth representation model, the local feature subspace reconstruction is re-executed, and the updated local reconstruction error is calculated. The updated local reconstruction error is used as the optimized detection result.

10. The deep learning-based big data anomaly detection system according to claim 1, characterized in that: The triggering of the corresponding early warning signal is specifically as follows: When the optimized test result is unqualified, the current time window is confirmed to be a true abnormal state, and the first warning signal is issued; A second warning signal is triggered when the access mode is a local anomaly mode and the optimized detection result is a failure. If the access mode is burst mode and the optimization detection result is qualified, no warning will be triggered.

Images