Big data anomaly detection method based on time sequence multi-dimensional auto-encoder

Abstract

The invention relates to the technical field of data processing, and further relates to a big data anomaly detection method based on a time sequence multi-dimensional auto-encoder. The method comprises the following steps of: 1, decomposing multi-dimensional time sequence data, extracting an internal oscillation mode of the multi-dimensional time sequence data by utilizing Hilbert-Huang transform, adaptively extracting multi-level space-time characteristics from the multi-dimensional time sequence data, and reconstructing the multi-level space-time characteristics to serve as a decomposition result; step 2, mapping a decomposition result into a low-dimensional and continuous manifold space; on the obtained manifold space, extracting structural information of the multi-dimensional time series data by calculating topological invariants such as continuous coherence; 3, calculating a local entropy of the multi-dimensional time sequence data on a set scale based on the extracted structural information; and calculating an abnormal probability according to the local entropy. According to the method, short-time sudden anomalies, long-term trend changes and multi-dimensional coupling anomalies can be efficiently captured.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and specifically relates to a big data anomaly detection method based on a time series multidimensional autoencoder. Background Art

[0002] With the continuous development of scenarios such as the Industrial Internet, the Internet of Things, and smart cities, massive multi-source heterogeneous data has exploded in various fields, especially in applications such as equipment status monitoring, production process control, financial risk control, and smart transportation. The acquisition of large-scale time series data is becoming more and more convenient. However, how to effectively identify potential anomalies in these massive data containing multidimensional, nonlinear, and non-stationary features and conduct risk warnings has become a topic of common concern in academia and industry. Traditional anomaly detection methods are mostly based on statistical analysis or machine learning techniques. Typical representatives include threshold detection methods based on single-dimensional statistics (such as mean, variance, skewness, etc.), dimensionality reduction detection based on principal component analysis (PCA), and classification models that rely on supervised or semi-supervised algorithms such as support vector machines and random forests. These existing technologies can indeed play a certain role in scenarios with small-scale, low-dimensional data, but their shortcomings are becoming increasingly apparent when facing the current multi-dimensional coupling, strong noise, and non-stationary big data time series.

[0003] Anomaly detection methods based on simple statistics or single thresholds are relatively ineffective for sudden and transient anomalies. For example, for sensor data of industrial equipment, some mechanical component failures may only show high-frequency jitter for a very short time, and then quickly return to the normal range; if you only rely on global statistics such as mean or variance, it is often difficult to capture such short-term anomalies in time. With the surge in the number of sensors and sampling frequency, the scale and dimension of data have increased significantly. Traditional detection methods based on Euclidean distance or single variable thresholds will be hit by the dual impact of "dimensionality disaster" and noise interference, resulting in a rapid decrease in detection accuracy and serious problems of false alarms and missed reports. Summary of the invention

[0004] The main purpose of the present invention is to provide a big data anomaly detection method based on time series multidimensional autoencoder, integrating adaptive time-frequency decomposition, wavelet packet transform, topological data analysis and local entropy measurement technology to accurately extract multi-scale features and topological structure information of multidimensional time series data. Through nonlinear dimensionality reduction and continuous homology analysis, the connectivity and hole distribution of multidimensional data in manifold space are characterized, and the local entropy and anomaly probability model are used to dynamically evaluate the system state. The present invention can efficiently capture short-term sudden anomalies, long-term trend changes and multi-dimensional coupling anomalies, significantly improving the sensitivity, accuracy and robustness of anomaly detection in complex environments.

[0005] In order to solve the above problems, the technical solution of the present invention is achieved as follows:

[0006] A big data anomaly detection method based on a time series multidimensional autoencoder, the method comprising:

[0007] Step 1: Decompose the multidimensional time series data. First, use the Hilbert-Huang transform to extract the intrinsic oscillation mode of the multidimensional time series data. Then, use the wavelet packet transform to capture the local features at different scales and time periods. Adaptively extract the multi-level spatiotemporal characteristics from the multidimensional time series data and reconstruct them as the decomposition result.

[0008] Step 2: Map the decomposition results into a low-dimensional and continuous manifold space; on the obtained manifold space, extract the structural information of the multidimensional time series data by calculating topological invariants such as continuous homology;

[0009] Step 3: Based on the extracted structural information, calculate the local entropy of the multidimensional time series data at the set scale; based on the local entropy, calculate the anomaly probability.

[0010] Furthermore, in step 1, the multidimensional time series data is assumed to be:

[0011]

[0012] Where M represents the dimension of multidimensional time series data; x m (t) is the m-th dimension of multidimensional time series data, m = 1, 2, ... M; T is the observation duration or the number of discrete sampling points; t is time; when the Hilbert-Huang transform is used to extract the intrinsic oscillation mode of multidimensional time series data, for each dimension x m (t) is processed as follows: x m (t) is decomposed into a set of intrinsic mode functions and a residual term:

[0013]

[0014] Among them, c m，k (t) is the kth intrinsic mode function of the mth dimension multidimensional time series data; K m is the total number of intrinsic mode functions of the m-th dimension multidimensional time series data; r m (t) is the residual term of the m-th dimension multidimensional time series data; for the k-th intrinsic mode function, the initial candidate intrinsic mode function is defined as:

[0015]

[0016] l is an integer subscript index; in the current candidate intrinsic mode function In the above example, we find all local maxima and local minima, and obtain the upper envelope by spline interpolation. With lower envelope The local average envelope is defined as:

[0017]

[0018] Update the candidate intrinsic mode function to:

[0019]

[0020] The standardized error is calculated as:

[0021]

[0022] When SD<∈, ∈ is the preset convergence threshold, it is considered The intrinsic mode function condition is satisfied, which is recorded as

[0023] Furthermore, in step 1, for each intrinsic mode function c m，k (t) The Hilbert-Huang transform is calculated by the following formula:

[0024]

[0025] Where pv represents the Cauchy principal value integral; τ is the time integral variable; the corresponding analytical multidimensional time series data z is constructed by the following formula m，k (t):

[0026]

[0027] Where j is an imaginary unit, satisfying j 2 = -1; is the instantaneous amplitude; is the instantaneous phase; is the instantaneous frequency.

[0028] Furthermore, in step 1, for each intrinsic mode function c obtained m，k (t) Wavelet packet transform is used to capture local features of different scales and time periods; let the mother wavelet be ψ(t), and construct the wavelet packet basis function corresponding to scale v and translation l:

[0029] ψ v，l (t) = 2 -v / 2 ψ(2 -v tl);

[0030] Among them, v∈{v min , v min +1, ..., v max} is the scale; v min and v max Represent the minimum scale and the maximum scale respectively; the intrinsic mode function c m，k (t) in the basis ψ v，l (t) and calculate the coefficient as local features.

[0031] Furthermore, in step 1, for the m-th dimension of multidimensional time series data, the reconstruction expression is:

[0032]

[0033] Among them, Λ v Represents the set of all valid translation indices under scale v; α m，k (v, l) is the reconstruction coefficient, which is calculated by taking into account the wavelet packet coefficient W m，k (v, l) and the instantaneous amplitude A extracted by the Hilbert-Huang transform m，k (t) and frequency ω m，k (t), defined as:

[0034] α m，k (v, l) = 0.3·W m，k (v, l)+0.4·A m，k (t)+0.3·ω m，k (t);

[0035] is the decomposition result of the m-th dimension multidimensional time series data. The decomposition result of the multidimensional time series data is

[0036] Furthermore, in step 2, the decomposition result is mapped to a low-dimensional and continuous manifold space by the following formula:

[0037]

[0038] in, The obtained manifold space is a smooth manifold; is the Kronecker product operation; ∥·∥ HS is the Hilbert-Schmidt norm; τ is the time integral variable; R(τ) is the Riemann curvature tensor.

[0039] Furthermore, in step 2, on the obtained manifold space, the structural information of the multidimensional time series data is extracted by calculating topological invariants such as continuous homology through the following formula:

[0040]

[0041] Where VR(M(X),∈) represents the Vietoris–Rips complex constructed based on M(X) at scale ∈; H m (VR(M(X),∈)) is the m-th dimensional homology group under scale ∈; γ m，i is the i-th homology class in the m-th dimension; b m，i With d m，iThey are γ m，i The birth scale and death scale of each barcode pair (b m，i d m，i ) is converted into numerical features; inf{·} is the lower bound operation; inf{∈:γ m,i vanishes} means that for a given homology class γ m，i , in all scales ∈ during the filtering process, such that γ m，i The infimum of those ∈ that disappear; For a given homology class γ m，i , find the minimum value of the scale parameter ∈ in the filtering process so that γ m，i First appeared in the m-th dimensional homology group H m (VR(M(X),∈)); ψ m (M(X)) is the structural information of the m-th dimension of multidimensional time series data under scale ∈.

[0042] Furthermore, in step 3, the structural information of the multi-dimensional time series data is assumed to be:

[0043] Ψ(M(X),∈)=[Ψ 1 (M(X),∈),Ψ 2 (M(X),∈),...,Ψ M (M(X),∈)];

[0044] Exponential normalization is applied to each component to define the normalized probability distribution:

[0045]

[0046] m=1,2,...,M;c=1,2,...,M;σ(∈) is the normalization factor under scale ∈, is the set value; using the normalized probability distribution p m (∈) Calculate the local entropy H(∈) at each scale ∈:

[0047]

[0048] Furthermore, in step 3, the abnormal probability is calculated according to the local entropy using the following formula:

[0049]

[0050] Among them, P anom is the abnormal probability.

[0051] The big data anomaly detection method based on time series multidimensional autoencoder of the present invention has the following beneficial effects:

[0052] The present invention provides a more sophisticated and adaptable overall solution for anomaly detection of multidimensional big data by integrating multiple methods such as adaptive time-frequency decomposition, wavelet packet segmentation and topological structure extraction on the basis of a multidimensional autoencoder, and achieves several obvious beneficial effects.

[0053] First, the present invention combines adaptive decomposition and local feature capture in the preprocessing stage of multi-dimensional time series data, so that the signals from different sensors or data channels are decomposed into multiple modes that can reflect different frequency bands and different time scales, and each mode is subjected to multi-level wavelet packet analysis and instantaneous amplitude and instantaneous frequency extraction. This method can accurately capture those short-term violent fluctuations or weak long-period interferences when faced with non-stationary, nonlinear and strong noise data, laying a high-resolution feature foundation for subsequent anomaly detection. Compared with traditional analysis schemes that only use fixed basis functions or are limited to a single dimension, the present invention can more adaptively reveal potential fault points or abnormal areas, while greatly reducing dependence on specific prior models.

[0054] Secondly, the present invention realizes nonlinear dimensionality reduction of high-dimensional feature vectors through a multidimensional autoencoder, projects complex multidimensional time series signals into a relatively low-dimensional and continuous space, and then uses topological structure extraction methods in the space to determine the overall morphology and local connectivity. Compared with simple linear dimensionality reduction methods or anomaly detection strategies based solely on distance metrics, the nonlinear dimensionality reduction adopted by the present invention can more fully retain the coupling relationship and nonlinear characteristics of the data itself, and with the introduction of the core idea of ​​topological data analysis, it can dynamically track the connected components, holes and ring structures of data in manifold space at different scales. When certain dimensions or subgroups show significant geometric deformation or topological fractures after dimensionality reduction due to anomalies, the algorithm can identify them in the multiscale filtering process, thereby greatly improving the sensitivity to complex anomalies and effectively reducing false positives and false negatives.

[0055] In addition, the present invention uses local entropy, an information theory indicator, to combine the uncertainty measure of multidimensional data in topological space with probability mapping, which can take into account two different types of anomalies: short-term rapid impact and long-term slow drift. Because local entropy can characterize the disorder and dispersion within the data in a multi-scale environment, when certain channels or features are outliers or the concentration increases abnormally within a specific time segment, the local entropy tends to change significantly. By mapping the entropy to an easily interpretable abnormal probability interval through an additional designed function, the present invention can not only provide intuitive quantitative results on the business side, but also avoid frequent false alarms due to local noise or slight fluctuations. This entropy and probability linkage strategy is both flexible and stable, and the steepness or offset of the mapping function can be fine-tuned according to the needs of different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 A method flow chart of a big data anomaly detection method based on a time series multidimensional autoencoder provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0058] Example 1, reference Figure 1 :A big data anomaly detection method based on a time series multidimensional autoencoder, the method comprising:

[0059] Step 1: Decompose the multidimensional time series data. First, use the Hilbert-Huang transform to extract the intrinsic oscillation mode of the multidimensional time series data. Then, use the wavelet packet transform to capture the local features at different scales and time periods. Adaptively extract the multi-level spatiotemporal characteristics from the multidimensional time series data and reconstruct them as the decomposition result.

[0060] In the specific implementation, the original multi-dimensional time series data will be firstly extracted through the screening process to gradually extract the intrinsic mode functions. These mode functions can respectively reflect the main change forms of the signal in different energy concentration intervals. At the same time, since this method does not rely on specific basis functions or preset signal assumptions, it has stronger adaptability to transient characteristics and nonlinear modes. After completing the separation of the intrinsic mode functions, the system will use the Hilbert transform to examine each mode as a whole from the perspective of the time axis and instantaneous frequency distribution. Through the linkage of instantaneous energy and instantaneous frequency, the frequency drift and amplitude change characteristics of each mode in different time segments can be more intuitively revealed. Since multi-dimensional time series data often comes from multiple sensors or multi-source data channels, it contains both trend information generated over time and mixed with coupling and correlation between different dimensions. Therefore, it is often difficult to accurately locate potential subtle anomalies or abnormal activities in a short period of time by only judging anomalies through original signals or single modes. Based on this, after obtaining each intrinsic mode function, the system will also perform multi-level wavelet packet decomposition on these mode functions in different frequency bands and time periods. As a more refined frequency band division method, wavelet packet decomposition can flexibly perform binary decomposition on the signal within the full frequency band, so that more refined components can be obtained in both high-frequency and low-frequency bands. In this way, each intrinsic mode can be further decomposed into more targeted sub-band information after wavelet packet decomposition, and a richer local feature description can be formed on the time-frequency plane.

[0061] Since any high-dimensional time series data often contains dynamic processes of different scales, for example, some long-period gentle changes may be related to the steady state or slow trend of the system, while short-period sharp fluctuations may indicate local sudden or abnormal conditions, wavelet packet decomposition can help capture such feature differences. When the intrinsic oscillation mode extracted by the Hilbert-Huang transform is combined with the advantages of wavelet packets in frequency domain segmentation, an adaptive and multi-scale analysis framework can be formed to provide a more comprehensive and in-depth feature representation for subsequent anomaly detection. At the same time, in the scenario of multidimensional data, the modal decomposition results of each channel or dimension can be separately subdivided by wavelet packets, and then combined with the correlation and synchronization analysis between multiple dimensions to obtain cross-dimensional spatiotemporal features, and finally these decomposition and reconstruction results are combined as an overall representation of multidimensional time series data at different time scales and frequency scales. This process helps to build more accurate low-dimensional mappings in subsequent autoencoder modeling and topological structure extraction, because when facing high-dimensional time series information, the autoencoder must rely on effective feature engineering to mitigate the dimensionality disaster and highlight potential abnormal pattern signals, and the multi-level spatiotemporal characteristics obtained through the above decomposition steps can provide high-resolution and high-discrimination inputs for subsequent dimensionality reduction and topological analysis. In other words, this decomposition and reconstruction idea is not just a simple signal preprocessing method, but by combining the Hilbert-Huang transform and the wavelet packet transform, it condenses the multiple modal characteristics of multi-dimensional time series data in time and frequency distribution, and realizes the dual characterization of small disturbances and overall trends at the cross-scale level.

[0062] Step 2: Map the decomposition results into a low-dimensional and continuous manifold space; on the obtained manifold space, extract the structural information of the multidimensional time series data by calculating topological invariants such as continuous homology;

[0063] In the specific implementation, since step 1 has extracted a reconstructed signal with rich spatiotemporal features for multidimensional time series data, when facing a scenario with high dimensionality and large data scale, directly using the original features for clustering or anomaly analysis often encounters dimensionality disaster or significant visualization difficulties. Through the dimensionality reduction capability of the autoencoder, the high-dimensional features can be first mapped to a latent space with lower dimensionality but still retaining the main structure and local geometric relationship. This latent space can be regarded as a manifold that is as continuous as possible, so that the originally complex data distribution can show a more analyzable form after dimensionality reduction. At this time, in order to further reveal the intrinsic topological structure of the data in this manifold space, it is necessary to use topological data analysis tools such as persistent homology to hierarchically characterize the connectivity, hole characteristics and higher-dimensional topological hole distribution of data samples at different scales. The calculation process of persistent homology can be understood as dynamically tracking the connection relationship between data points when gradually increasing or decreasing a certain neighborhood scale, observing how the connected components are born and die, and recording the appearance and disappearance of ring structures or cavities at different stages, thereby forming a set of barcodes or persistence graphs reflecting the topological evolution process. For big data anomaly detection based on time series multidimensional autoencoders, these topological invariants can largely characterize the global shape and local sparsity of the data, helping to identify abnormal distributions that are very different from normal patterns. Compared with traditional anomaly detection methods that simply rely on distance metrics or density estimation, topological data analysis can break away from the strict reliance on Euclidean metrics, start more from the global structure, and capture potential nonlinear distribution characteristics, thereby making up for the defects of ordinary distance metrics in high-dimensional complex scenarios that are easily disturbed by noise or scale changes. When some channels in multidimensional time series data show outlier behavior or drastic fluctuations in frequency distribution during special periods, these outliers or abnormal clusters are likely to have topological differences from the original normal pattern in the manifold space, such as forming new independent connected components, or causing prominent holes or branch deformations in the originally smooth low-dimensional manifold. By calculating and comparing these persistent homology invariants, the system can more accurately capture the degree of deviation of anomalies at the topological level without relying on explicit probability distribution assumptions. At the same time, the nonlinear dimensionality reduction strategy implemented by the autoencoder can also make the construction of the manifold more consistent with the complex coupling relationship of the multidimensional time series data itself, avoiding feature distortion or information loss caused by simple linear dimensionality reduction. Since the data is easier to visualize after dimensionality reduction, researchers or system managers can also intuitively evaluate the location of the anomaly, the topological properties of the surrounding neighborhood, and the evolution trajectory of the anomaly at different scales based on the performance of the persistence graph or barcode graph.

[0064] Especially in the context of big data, this approach of combining autoencoders with topological data analysis can mine global geometric features that are difficult to detect by distance measurement while maintaining high parallelism and large scale. In summary, what step 2 does is to use the reduced-dimensional manifold space to construct a suitable topological perspective, so that the reconstructed spatiotemporal features are no longer just a series of abstract vectors or matrices, but have interpretable shapes, holes, and connectivity distributions on the manifold. At the same time, the multi-scale analysis properties of continuous homology can cover a series of changes from microscopic local to macroscopic overall. The persistent features extracted from them are often robust to noise, and can give significant difference characterization of abnormal spatial distribution, and also provide a solid structural foundation for the subsequent probability evaluation in combination with local entropy in step 3. In actual deployment, in order to obtain the most accurate dimensionality reduction and topological characterization effects, it is necessary to make reasonable parameter selection based on the data scale, the characteristics of modal decomposition, and the network structure of the autoencoder, and combine iterative training and cross-validation to ensure that the reconstruction error after manifold embedding is within an acceptable range. When the system monitors a new time series in the data stream, it will quickly project it into the learned low-dimensional manifold and observe its topological relationship with the main group in the continuous homology graph. This makes online detection or semi-real-time alarm possible, effectively improving the efficiency of industrial control systems or Internet of Things platforms in identifying potential risks.

[0065] Step 3: Based on the extracted structural information, calculate the local entropy of the multidimensional time series data at the set scale; based on the local entropy, calculate the anomaly probability.

[0066] Since the high-dimensional, multimodal time series data has been mapped to a low-dimensional manifold space in step 2, and several topological invariants that can represent the global and local structural differences of the data have been obtained by means of continuous homology analysis, in this step, the system needs to further combine the idea of ​​entropy measurement with the distribution of these topological features in different regions to measure the complexity or disorder of the data in the local neighborhood. When the data distribution in a sub-region is relatively uniform and structurally stable, it usually means that it maintains good coherence with the main group or normal mode on the manifold, and its corresponding local entropy value is often small; on the contrary, if some data points or sub-regions show signs of being outliers, scattered, extremely sparse, or even completely disconnected from the surrounding areas, then the neighborhood where these points are located is likely to deviate significantly from the normal interval in topological structure, and will show a higher degree of disorder or complexity, and the corresponding local entropy value may surge. By comparing these local entropies with the pre-set discrimination threshold or statistical model, the degree of abnormality of a certain time series at the current scale can be quantitatively evaluated. Furthermore, since multidimensional time series data usually have multiple time scale characteristics, the calculation of local entropy can also be performed for different time windows or frequency bands, so that both long-term stable trends and short-term sudden changes can be taken into account.

[0067] When the time window is large, the system focuses on the overall structural stability. If some channels or modes continue to deviate from the normal state over a long time scale, the corresponding local entropy values ​​will be repeatedly high in multiple iterative calculations, indicating that there may be abnormal signs of slow evolution; when the time window is small, it pays more attention to the response to transient or subtle disturbances. Especially in industrial control or IoT monitoring scenarios, such short-term and high-frequency changes often indicate potential failures or emergencies, which require timely detection and early warning. By integrating the calculation results of local entropy with topological structure analysis and dimensionality reduction information of autoencoders, the system can more accurately identify local areas that are isolated or abnormally clustered in the manifold space. In order to improve robustness, the actual deployment will also combine sliding windows, exponential weighting and other methods to allow the local entropy index to be dynamically updated over time. This can not only continuously track the evolution of the overall distribution during the slow drift of data, but also capture those sudden jumps. When certain channels or patterns frequently experience dramatic entropy fluctuations in a short period, or maintain a high entropy state for a long period, the system will mark these situations as high risk and then determine whether to perform further diagnostic analysis based on other conditions.

[0068] It is worth noting that, unlike the traditional global entropy or simple probability distribution-based anomaly detection method, the local entropy calculation adopted by the present invention is based on the neighborhood structure on the manifold, and at the same time it echoes with the previous continuous homology information, which means that the discrimination criterion does not only depend on whether the density or global distribution conforms to a certain hypothesis, but starts from the topological and spatiotemporal characteristics of the data itself, giving the algorithm a higher sensitivity to nonlinear and non-stationary features. In many application scenarios involving multi-sensor fusion or multi-source heterogeneous data, this anomaly detection method based on topology and local entropy is more likely to find those subtle distribution changes that are difficult to capture by conventional mathematical and statistical methods, thereby effectively reducing the risk of underreporting and improving the overall detection accuracy. It is particularly worth emphasizing that the autoencoder provides a dimensionality reduction channel with nonlinear representation capabilities in step 2, so that the manifold space can better fit the multidimensional coupling relationship of the actual data, laying a solid representation foundation for the calculation of local entropy. Without the nonlinear mapping of the autoencoder, simple linear dimensionality reduction often causes potential abnormal patterns to be distorted or superimposed on each other during the mapping process, making it difficult to obtain clear differential expressions in subsequent entropy measurements. At the same time, in a big data environment, this method can also use parallel computing and distributed storage to complete anomaly detection of all nodes or time segments in the global scope through batch evaluation of local entropy values. Even if the data scale is large and the dimensions are numerous, it will not become infeasible due to excessive computing overhead.

[0069] Example 2: In step 1, the multi-dimensional time series data is assumed to be:

[0070]

[0071] Where M represents the dimension of multidimensional time series data; x m (t) is the m-th dimension of multidimensional time series data, m = 1, 2, ... M; T is the observation duration or the number of discrete sampling points; t is time; when the Hilbert-Huang transform is used to extract the intrinsic oscillation mode of multidimensional time series data, for each dimension x m (t) is processed as follows: x m (t) is decomposed into a set of intrinsic mode functions and a residual term:

[0072]

[0073] Among them, c m,k (t) is the kth intrinsic mode function of the mth dimension multidimensional time series data; K m is the total number of intrinsic mode functions of the m-th dimension multidimensional time series data; r m (t) is the residual term of the m-th dimension multidimensional time series data; for the k-th intrinsic mode function, the initial candidate intrinsic mode function is defined as:

[0074]

[0075] l is an integer subscript index; in the current candidate intrinsic mode function In the above example, we find all local maxima and local minima, and obtain the upper envelope by spline interpolation. With lower envelope The local average envelope is defined as:

[0076]

[0077] Update the candidate intrinsic mode function to:

[0078]

[0079] The standardized error is calculated as:

[0080]

[0081] When SD<∈, ∈ is the preset convergence threshold, it is considered The intrinsic mode function condition is satisfied, which is recorded as

[0082] Specifically, the multidimensional time series data is recorded as X(t), where each dimension corresponds to an x m (t), we can iteratively screen each dimension separately, and m (t) is decomposed into the sum of several intrinsic mode functions, and a residual that tends to be smooth or low-frequency is left at the end. The theoretical support behind this approach is: in many real scenarios, signals are often mixed with multiple levels of fluctuations and trends. If we only rely on fixed basis functions or linear assumptions to separate high and low frequencies, we will often miss some key burst modes or non-stationary components; therefore, the present invention uses this decomposition formula and the adaptive characteristics of the Hilbert-Huang transform to better capture the subtle but short-term changes in complex systems that may cause abnormalities. When writing , the left side of the equal sign represents the original time series signal of the mth dimension, the first half of the right side is the sum of the multiple layers of intrinsic mode functions extracted in sequence, and the second half is the residual term that has not been explained by any mode function. This decomposition form is suitable for multi-dimensional time series scenarios because it can use the same idea to screen different dimensions separately, and then compare and combine all mode functions in the same multi-dimensional framework in subsequent analysis.

[0083] In the specific screening step, the sum of the previously extracted modal functions is first removed from the original signal to obtain an initial candidate function Then, the upper and lower envelopes are constructed by local extreme value search and spline interpolation, and their average is calculated to obtain the local average trend of the candidate function in the current iteration. The reason for performing such envelope subtraction is to ensure that the final intrinsic mode function satisfies the characteristics of oscillation center symmetry and avoid residual obvious trend components or asymmetric distribution. At the same time, in order to measure the amplitude of function change before and after each iteration, the algorithm uses standardized error to determine whether it has converged. As long as its value is lower than a certain threshold ∈, it means that the current candidate function meets the definition requirements of the intrinsic mode and no longer needs to be screened. Repeat this process until the new oscillation mode can no longer be separated, and a complete set of modal functions and a relatively smooth residual are obtained. Because in real big data scenarios, time series signals may show a variety of complex forms such as rapid oscillation, slow rise, and burst pulses. If they are not separated first, the subsequent frequency domain analysis and anomaly detection process will often encounter a lot of interference. Through this decomposition method, the present invention can extract multiple subdivided fluctuation components for each dimension, which not only provides a high-resolution foundation for subsequent multi-scale analysis based on wavelet packets, but also lays a reference for the coupling relationship between multiple dimensions in different time segments. For example, some modal functions may have violent peak-to-valley fluctuations in the high-frequency range, corresponding to some short-term abnormal events, while other modal functions show more stable long-period characteristics, suggesting a slow evolution process that may contain systemic risks; when comparing across dimensions, if several dimensions have synchronous pulses or coupled fluctuations in the same modal sequence, it may mean that there is a linkage effect between multiple subsystems, which has great diagnostic value for large-scale industrial monitoring or Internet of Things platforms. Furthermore, the present invention uses such modal decomposition results to input the autoencoder so that when constructing a low-dimensional manifold space, it can focus on the differences between the modes and capture the global structure through persistent coherence, ultimately making the anomaly detection of local entropy calculation more targeted and robust. In short, by splitting the original time series data into several independent oscillation components and a smooth residual in each dimension, these formulas demonstrate the adaptability of the Hilbert-Huang transform to non-stationary signals on the one hand, and on the other hand, they can also provide sufficiently delicate underlying feature support for practical applications such as industrial control and fault warning. Without the constraints of prior basis functions, these modal functions adaptively map the multiple changes of the signal in the time domain and frequency domain, and perfectly connect with the subsequent multi-dimensional feature aggregation and autoencoder dimensionality reduction. At the same time, the existence of the residual term makes it possible to capture long-term trends or fixed offsets that may be ignored, thereby achieving a more comprehensive characterization of the overall system operation health.

[0084] Example 3: In step 1, for each intrinsic mode function c m，k (t) The Hilbert-Huang transform is calculated by the following formula:

[0085]

[0086] Where pv represents the Cauchy principal value integral; τ is the time integral variable; the corresponding analytical multidimensional time series data z is constructed by the following formula m，k (t):

[0087]

[0088] Where j is an imaginary unit, satisfying j 2 = -1; Instantaneous amplitude; is the instantaneous phase; is the instantaneous frequency.

[0089] Specifically, for any intrinsic mode function c obtained from the empirical mode decomposition m，k (t), first construct its corresponding Hilbert transform through some kind of integral operation, denoted as H{c m，k}(t), where the concept of the Cauchy principal value (i.e., pv) is used to deal with the singularity of the integral kernel at t-τ=0, thereby avoiding the integral being mathematically non-convergent or not satisfying the absolute integrability condition in the usual sense. Precisely because multidimensional time series data often exhibits complex characteristics such as non-stationarity, strong noise, and nonlinearity in real industrial or IoT scenarios, the Hilbert transform method can construct an approximate analytical signal for each modal function, so that there is a relatively independent but related correspondence between the real and imaginary parts of the mode at the same time. Treat the original modal function as the real part, the result of the Hilbert transform as the imaginary part, and then use addition to obtain a form of c m，k (t)+jH{c m，k}(t), we can rewrite it into the exponential form of amplitude and phase with the help of Euler's formula, thus obtaining the instantaneous amplitude A m，k (t) and instantaneous phase θ m，k (t). The amplitude is obtained by taking the square root of the real part plus the square root of the imaginary part, and the phase comes from the inverse tangent function, which represents the local oscillation position of the modal function at each time point. When the instantaneous phase is derived with respect to time, the instantaneous frequency ω can be obtained. m，k (t), which means that at each moment, the oscillation speed of the intrinsic mode function can be measured with a more continuous frequency index. Since similar transformations are performed on each dimension of multidimensional time series data, the instantaneous amplitudes and instantaneous frequencies of different dimensions can be compared in parallel in the same time segment to determine whether there is synchronous peak fluctuation or frequency drift.

[0090] If several dimensions are obviously coupled in instantaneous amplitude or instantaneous frequency, it often implies that they are related in some way in the underlying physical or system mechanisms. This is an extremely important clue in big data anomaly detection, because some fault signs often come from abnormal states presented by multiple subsystems or sensors. At the same time, by analyzing the signal form, it is also possible to more intuitively track whether certain modes have drastic phase jumps or amplitude mutations in special periods of time. These jumps often correspond to possible abnormal outbreak points. If combined with subsequent wavelet packet multi-scale analysis and autoencoder dimensionality reduction, these instantaneous amplitudes and frequency information can be mapped to the manifold space together, and the topological structure and local entropy can be used to further determine the anomaly. Compared with global Fourier transforms in the frequency domain or short-time Fourier analysis that relies on a fixed window length, the instantaneous parameters of the Hilbert transform can more fully reflect the dynamic evolution of the signal on the time axis, and are especially suitable for non-stationary or multi-component overlapping data. In industrial sites, some faults may only cause sharp frequency drifts in a very short time window. Traditional algorithms based on global average energy or single frequency band statistics often ignore these details. In the present invention, by analyzing the instantaneous amplitude and instantaneous frequency of each intrinsic modal function, the inherent laws of these short-term mutations can be presented to the greatest extent. Furthermore, small fluctuations in the instantaneous frequency of a single dimension in a multidimensional scenario may not be sufficient to explain the anomaly, but if certain modes have synchronous increases or decreases in instantaneous frequency in multiple dimensions at the same time, they can be identified as high-risk signals by subsequent autoencoders. In actual applications, this also helps the automated monitoring system to realize alarm integration for multi-source data, which not only reduces the probability of single-point false alarms, but also takes into account the multi-dimensional dependency structure in a complex environment.

[0091] Example 4: In step 1, for each intrinsic mode function c obtained m，k (t) Wavelet packet transform is used to capture local features of different scales and time periods; let the mother wavelet be ψ(t), and construct the wavelet packet basis function corresponding to scale v and translation l:

[0092] ψ v，l (t) = 2 -v / 2 ψ(2 -v tl);

[0093] Among them, v∈{v min , v min +1, ..., v max} is the scale; v min and v max Represent the minimum scale and the maximum scale respectively; the intrinsic mode function c m，k (t) in the basis ψ v，l (t) and calculate the coefficient as local features.

[0094] Specifically, we first select a mother wavelet function ψ(t), and then expand it by scaling and translation to construct a set of wavelets of the form ψ v，l (t) basis function, where v and l represent the expansion and movement of the wavelet packet in the time and frequency dimensions respectively. Since the actual data scale is often extremely large and contains many non-stationary and nonlinear characteristics, it is often difficult to obtain the same fine decomposition in all sub-bands if only relying on a single layer or simple discrete wavelet transform; while the wavelet packet can continue to split into two branches in each frequency band, not limited to only refining the low-frequency part, and then obtain the uniform subdivision ability of the entire frequency band at the multi-scale level, which is very beneficial for capturing sudden events and weak periodic oscillations. Specifically, when a certain intrinsic mode function c m，k (t) is projected onto ψ v，l (t) When the function family is formed, we can get some coefficients W m，k (v, l), the size of the coefficient reflects the energy distribution of the modal function in a given scale and time period, and also reveals its transient characteristics that may be hidden in a specific local window. For example, if a coefficient peak much higher than the adjacent interval appears in a higher frequency band (corresponding to a small large-scale parameter v), it indicates that the mode has significant high-frequency disturbances during this period, and may be identified as a potential risk point in the subsequent anomaly detection process; if multiple energy-concentrated fragments continue to appear in a lower frequency band (corresponding to a large v), it may also mean some kind of long-term oscillation or slow decay phenomenon. Similarly, the time displacement effect brought by the translation parameter l allows the algorithm to scan the time axis of the entire signal more flexibly, which is particularly critical in non-stationary signals, because many anomalies do not occur at fixed times or fixed frequency bands, but appear in different intervals as the system state or external environment changes.

[0095] By observing the wavelet packet coefficients, we can often clearly depict c in the time-frequency plane. m，kThe dynamic structure of (t) provides highly distinguishable feature vectors for the subsequent dimensionality reduction combined with the autoencoder. Compared with traditional wavelet decomposition, the biggest feature of wavelet packets is the high resolution and balanced coverage brought by the full-band binary decomposition, which is crucial for the multi-dimensional big data scenario faced by the present invention. Because when the modal function coefficients of multiple dimensions are merged into the same analysis framework, the same or similar scale parameter v can be used to compare the local energy distribution of different dimensions, so as to determine whether there are coupled fluctuations or synchronization anomalies in multi-source signals on similar frequency bandwidths; if some dimensions have abnormal surges in coefficients in specific sub-bands, it is very likely caused by systemic failures or emergencies. These suspicious time periods can be focused on monitoring or marked as high risk in the manifold space. At the same time, this multi-scale feature can also improve the training performance of subsequent autoencoders: because the network can gradually capture the nonlinear combination relationship between the coefficients of each sub-band during the learning process, it is easier to show the unique topological structure of abnormal patterns in the encoded low-dimensional potential representation. It is worth emphasizing that the wavelet packet coefficients can not only identify different signal intervals in terms of scalar size, but also complement the instantaneous amplitude, instantaneous frequency and other features of the previously analyzed signal to form a multi-angle and multi-scale feature fusion. In this way, even in an extreme noise environment, the algorithm can rely on the high-dimensional vectors aggregated from different features to maintain a strong degree of distinction, thereby minimizing sensitivity to single noise or local sampling inaccuracy.

[0096] Example 5: In step 1, for the m-th dimension of multidimensional time series data, the reconstruction expression is:

[0097]

[0098] Among them, Λ v Represents the set of all valid translation indices under scale v; α m,k (v,l) is the reconstruction coefficient, which is calculated by taking into account the wavelet packet coefficient W m,k (v,l) and the instantaneous amplitude A extracted by the Hilbert-Huang transform m,k (t) and frequency ω m,k (t), defined as:

[0099] α m，k (v, l) = 0.3·W m,k (v, l)+0.4·A m，k (t)+0.3·ω m，k (t);

[0100] is the decomposition result of the m-th dimension multidimensional time series data. The decomposition result of the multidimensional time series data is:

[0101]

[0102] Specifically, although the traditional wavelet packet transform can effectively capture the local energy distribution of each scale and time period, if it only relies on simple wavelet packet coefficients, it may not be able to fully characterize the transient patterns or short-term mutations that appear in non-stationary and nonlinear environments; and the instantaneous amplitude and instantaneous frequency obtained in the Hilbert-Huang transform stage can more flexibly describe the specific rate and amplitude of dynamic changes of different modal functions over time. Based on this, the present invention proposes to let the reconstruction coefficients refer to these three elements-wavelet packet coefficients, instantaneous amplitude and instantaneous frequency at the same time, and assign different weight coefficients to them. In this way, each projection under the wavelet packet basis function not only reflects the local energy information, but also carries a response to the instantaneous behavior characteristics, which can more fully express the original multidimensional sequence in the short-term high-frequency or long-term low-frequency interval during signal reconstruction. The multi-layer characteristics, thereby providing more accurate input for subsequent dimensionality reduction and anomaly discrimination. Specifically, each intrinsic modal function is in the wavelet packet basis function ψ v，l The coefficients obtained under (t) can be first regarded as coarse-grained local energy measures. These coefficients are combined with the instantaneous amplitude and frequency obtained from the Hilbert-Huang transform and finally gathered into the reconstruction coefficient α m，k (v, l), and the reconstruction coefficient is the key to determine the contribution of the modal function to the overall signal under a specific scale v and translation l. When a modal function has a very large amplitude at an instant, or its frequency rises sharply, it means that the signal may have more drastic fluctuations or abnormal precursors during this period, then the reconstruction coefficient of the corresponding position will be amplified by weighting, highlighting the importance of these potential abnormal fragments in the entire reconstruction process; conversely, if some modes only maintain a small fluctuation amplitude at a certain scale, then their reconstruction coefficient will be relatively smaller, thus giving them a lower weight during reconstruction.

[0103] After allocating the coefficients under all scales and translation indices, the sub-band components can be superimposed on the wavelet packet basis functions to achieve a balance between the time-frequency analysis results at different levels, which not only retains the weak but potentially important transient information, but also does not over-amplify the accidental peaks caused by simple noise. m(t) plays the role of preserving the long-term or global trends that can no longer be precisely characterized by intrinsic mode functions or wavelet packet basis functions. This is particularly critical in scenarios where certain systemic faults evolve slowly. Many times, faults are not only reflected in local instantaneous fluctuations, but may also accumulate slowly over a long period of time. If this trend component is not retained, the system may ignore some low-frequency but continuous danger signals. The present invention retains the residual during the final reconstruction to ensure that the subsequent autoencoder input contains not only diverse oscillations, but also possible overall translations or slow fluctuations, thereby meeting the need to give equal weight to global and local features of non-stationary data. In multidimensional scenarios, each dimension can be obtained through this reconstruction formula. Then put all Combined to form reconstruction data for the entire system Since these data include wavelet packet energy distribution, Hilbert instantaneous parameters and residual trends, subsequent autoencoders can more accurately capture complex patterns such as coupling effects, phase synchronization and frequency resonance between multidimensional data when faced with such rich inputs, thereby retaining the key information that is most sensitive to anomalies when reducing the dimension.

[0104] Example 6: In step 2, the decomposition result is mapped to a low-dimensional and continuous manifold space by the following formula:

[0105]

[0106] in, The obtained manifold space is a smooth manifold; is the Kronecker product operation; ||·|| HS is the Hilbert-Schmidt norm; τ is the time integral variable; R(τ) is the Riemann curvature tensor.

[0107] Specifically, in order to compactly embed these high-dimensional features into a low-dimensional manifold, we need to use the idea of ​​differential geometry to To modify the space in which the feature is located, it is necessary to define a mapping that is linked to a certain curvature structure so that the features collected at different time points can reflect local similarity while maintaining a certain geometric coherence as a whole. In this way, the exponential function in the formula plays an important role in stretching or compressing. By Kronecker product (i.e. The result of multiplying the result of the operation) with its own transpose is converted into a matrix that can reflect the coupling relationship between the components, and then the Hilbert-Schmidt norm is used The purpose of standardizing it is to maintain a relatively balanced scale between time segments of different magnitudes. This can prevent certain component values ​​from being too large and monopolizing the overall geometric structure in a big data environment, and can also ensure that those weak but important abnormal signals are not completely submerged during mapping. Next, through exponential mapping, the matrix is ​​mapped to a structure related to the Lie group, and then it can be multiplied with the Riemann curvature tensor R(τ) to integrate the result that was originally just a characteristic inner product or outer product into a differentiable manifold measurement framework. In other words, τ is both a variable of the time series integral and a link between the entire topological structure and the manifold geometry, because at each moment can all be regarded as a tiny fragment on this manifold, and R(τ) combines these fragments into a continuous, internally coherent geometric body by applying differentiated curvature constructions at different time fragments. When integrating time from 0 to T, it is equivalent to scanning all the features in the entire observation window, accumulating them through exponential mapping and curvature modulation, and shaping a low-dimensional space that can reflect the relationship between time evolution and multidimensional coupling. This space is referred to as M(X) in the formula. Once M(X) forms a smooth manifold, it means that the geodesic distance between different points can be calculated on it, the connectivity of the local neighborhood can be tracked, and topological data analysis methods such as continuous homology can be used to identify whether there are abnormal clusters, void structures, or other topological features that are significantly different from the normal distribution.

[0108] More importantly, this construction method allows orderly geometric integration of high-dimensional features in a big data environment. Compared with the traditional dimensionality reduction method based on Euclidean metric, it can more keenly capture key factors such as nonlinear distribution, instantaneous coupling, and multi-scale interaction in time and space, thereby bringing higher accuracy and reliability to the anomaly detection process of the present invention. Especially in complex industrial control or Internet of Things monitoring scenarios, different sensor data often present intricate coupling relationships at the same time. If only simple linear mapping is used to reduce the dimension, many subtle but potentially important patterns will be ignored; while this method uses the Kronecker product to expand the interactive description surface of each component, and then regularizes it through the Hilbert-Schmidt norm to prevent numerical explosion or deformity caused by the principal component effect. Finally, the Riemann curvature tensor is coupled to the mapping process, so that these feature components form a sufficiently smooth, slightly bendable and tolerant abnormal mutation geometric structure in the time dimension. It is precisely because this mapping step provides a coherent and fully expressive manifold for subsequent continuous homology that possible anomalies can be quickly identified in the changes of local neighborhoods or multi-scale holes. If the reconstructed data of a certain period or certain dimensions have discontinuous branches, isolated connected components or significant deformation on the manifold, it often means that the system is or will be abnormal. At the same time, in the context of big data, the integral form in the formula is also convenient for batch parallel computing, because multiple time periods can be locally summed and then merged into an overall manifold; as long as the and ||·|| HS With a unified definition of and effective storage of R(τ), the manifold mapping of massive features can be completed in a distributed manner.

[0109] Example 7: In step 2, on the obtained manifold space, the structural information of the multidimensional time series data is extracted by calculating topological invariants such as continuous homology through the following formula:

[0110]

[0111] Among them, VR(M(X),∈) represents the The constructed Vietoris–Rips complex; H m (VR(M(X),∈)) is the m-th dimensional homology group under scale ∈; γ m，i is the i-th homology class in the m-th dimension; b m，u With d m，i They are γ m，i The birth scale and death scale of each barcode pair (b m，i , d m，i ) is converted into numerical features; inf{·} is the lower bound operation; inf{∈:γ m，i vanishes} means that for a given homology class γm，i , in all scales ∈ during the filtering process, such that γ m，i The infimum of those ∈ that disappear; For a given homology class γ m，i , find the minimum value of the scale parameter ∈ in the filtering process so that γ m，i First appeared in the m-th dimensional homology group H m (VR(M(X),∈)); is the structural information of the m-th dimension of multidimensional time series data under scale ∈.

[0112] Specifically, after a data set M(X) embedded on a manifold has been obtained through the previous steps, the corresponding Vietoris–Rips complex, namely VR(M(X), ∈) can be constructed under different scale parameters ∈. The so-called Vietoris–Rips complex means that under a given distance metric, as long as the distance between any two points does not exceed ∈, they will be connected by an edge in the complex; for clusters of three, four or even more points, simplified shapes (such as triangles, tetrahedrons, etc.) will be added layer by layer according to whether the distances between them are within the range of ∈. By gradually increasing ∈ from a minimum value, this complex will undergo a filtering process from discreteness to the continuous addition of connected components, ring structures and even higher-dimensional voids; in topological data analysis (TDA), each scale increment corresponds to a "filter", and the system will track the homology groups H in different dimensions. m How new homology classes are generated and how they disappear at a larger scale. Therefore, the topological features that appear on a certain dimension m can be recorded as the corresponding homology class γ m，i , and their first appearance scale and final extinction scale in the filtering process are called birth scale b and b respectively. m，i and the extinction scale d m，i .

[0113] In the formula used in the present invention, inf{∈:γ m，i ∈H m (VR(M(X),∈))} represents the smallest ∈ where the homology class first appears, and inf{∈:γ m，i disappears} corresponds to the smallest ∈ that is filled or merged and disappears during the filtering process. m，i , d m，i) statistics or visualization, we can get the so-called barcode expression, marking the interval of birth and death on the horizontal axis, and each line segment corresponds to the life cycle of a topological feature; those features with very short lifespans are usually regarded as noise or local disturbances, while homology classes with longer lifespans often mean that the data has significant connected components, rings or hollow structures within a certain scale range. It is precisely under this distinction that the system can identify the global and local geometric distribution differences of multidimensional time series data, and then connect them with the manifold structure after dimensionality reduction based on the time series multidimensional autoencoder to discover abnormal patterns.

[0114] For example, for multi-sensor time series in industrial processes, when ∈ is very small, if the data points are sparsely distributed on the manifold, a large number of zero-dimensional homology classes, that is, many independent connected components, often appear; as ∈ increases, these components will connect to each other, and the zero-dimensional homology classes will gradually merge. If some components are unable to connect with the mainstream components, or even remain isolated after a considerable ∈, it is likely to correspond to some outlier behaviors or abnormal clusters. In one-dimensional homology, if a stable ring structure is formed as the filtering process evolves, it means that the data has a relatively significant cyclic or periodic distribution in this dimension; if a one-dimensional homology class appears and maintains a long life cycle in a certain period of time or in a subcluster, it generally indicates some significant internal feedback mechanism or abnormal cycle mode of the system. For higher-dimensional homology (such as two-dimensional and three-dimensional), it may be related to holes, cavities or more complex structures of the data in the manifold space. In the big data scenario, this method of constructing Vietoris–Rips complexes and performing multi-scale filtering can adapt well to the distribution of high-dimensional and large-scale samples: even if the amount of data is extremely large, as long as the point-to-point distance can be efficiently calculated in a distributed architecture, the homology information can be continuously updated in an incremental or parallel manner, without being troubled by excessive noise or scale parameter selection like traditional outlier analysis based on Euclidean distance or probability models. At the same time, continuous homology also provides a systematic characterization of multi-resolution structures, so that fine patterns that may be ignored at very small ∈ can gradually appear as ∈ increases; if these fine patterns are detected to still survive at a relatively large scale, they can be judged to be of sufficient importance and thus marked as potential anomalies.

[0115] In order to digitize such topological results, the present invention also performs a m，i , d m，i ) to convert several characteristic indicators that can be used for machine learning or statistical analysis. For example, its length d can be directly recorded. m，i -b m，i, some kernel function mapping can also be performed on the barcodes to synthesize a topological signature of a collection of different barcodes, so as to incorporate topological structure differences in the next step of local entropy calculation or classification discrimination. If the barcodes of some homology classes are abnormally long, or a group of features appear in the same time window, it can be suggested that the system has unusual behavior in this period or this component combination, and further inspection is necessary. In addition, these topological invariants can also be combined with the instantaneous amplitude, instantaneous frequency or wavelet packet energy distribution generated by decomposition and reconstruction to form a multi-angle anomaly metric, thereby ensuring that anomaly identification is neither limited to local changes in a single dimension nor only focuses on the global average distribution, but is aware of the geometric deformation of the entire signal at different scales and dimensions. It is worth mentioning that this topological analysis driven by persistent coherence complements the deep dimensionality reduction of the autoencoder: the autoencoder provides a way to map high-dimensional data to a compressible manifold space that retains structural information as much as possible, and the multi-scale filtering process of persistent coherence further reveals how the manifold is connected or decomposed at different radii, so that the abnormal signals hidden behind the multi-dimensional coupling or nonlinear relationship can be amplified and highlighted. For industrial control systems, Internet of Things monitoring platforms or various complex big data applications, this method can be deployed in practice by first sending the time series data to the autoencoder in segments based on a block or sliding window method to obtain the corresponding low-dimensional representation, and then forming a barcode on these low-dimensional representations by constructing the Vietoris–Rips complex and calculating the coherence group. After a batch of barcodes are generated in each period of time, the pattern differences of these barcodes in different time periods are compared, or the coherence information of adjacent intervals is merged to identify whether there is a persistent anomaly. If some abnormal topological features are persistently significant in the time-frequency domain, they can be judged as high-risk events by the system and automatically trigger alarms or start fault-tolerant strategies, thereby improving the coverage of short-term faults or slow-evolving problems. It can be seen that the formula given in Example 7 describes a complete multi-dimensional topology extraction process: first, through the filtering process of the Vietoris–Rips complex under different ∈, track H m Homology class in the group m，i How to be born and die, and will be born inf{∈:γ m，i ∈H m (·)} and the extinction inf{∈:γ m，i Disappear} Record the two items and then map them into (b m，i , d m，i ) in the form of a barcode pair or other numerical signature, and then regard this information as Ψ mThe multi-dimensional time series structural features encapsulated by (M(X),∈). Such structural features can be said to extend the measurement method that tends to be point-to-point or local neighborhood in traditional signal processing to the global geometric level, allowing the algorithm to perceive potential anomalies from the "shape" rather than being limited to linear statistics such as mean and variance. When used in conjunction with the dimensionality reduction results based on the time series multi-dimensional autoencoder, it is possible to accurately distinguish on the manifold whether there are abnormal topological deformations such as independent components, rings or holes, thereby improving the anomaly detection ideas of the present invention in big data nonlinear scenarios and ensuring its higher sensitivity.

[0116] Example 8: In step 3, the structure information of the multi-dimensional time series data is assumed to be:

[0117] Ψ(M(X),∈)=[Ψ 1 (M(X),∈),Ψ 2 (M(X),∈),...,Ψ M (M(X),∈)];

[0118] Exponential normalization is applied to each component to define the normalized probability distribution:

[0119]

[0120] m=1,2,...,M;c=1,2,...,M;σ(∈) is the normalization factor under scale ∈, is the set value; using the normalized probability distribution p m (∈) Calculate the local entropy H(∈) at each scale ∈:

[0121]

[0122] Specifically, in Example 8 of the big data anomaly detection method based on time series multidimensional autoencoder, by exponentially normalizing the topological structure information on the aforementioned manifold space and calculating the local entropy, the complex distribution and abnormal risk of multidimensional time series data at different scales can be more delicately portrayed. When analyzing the structural information of multidimensional time series data, it is often necessary to comprehensively consider the connectivity, void characteristics, and barcode performance obtained in the continuous homology analysis of different dimensions on the manifold, so it is summarized into a vector form of Ψ(M(X),∈), where each component Ψ m (M(X), ∈) represents the numerical description of a dimension or a topological feature at scale ∈. In order to facilitate the subsequent combination of the difference measurement between different components with the information theory method, these values ​​need to be normalized. The present invention defines p by exponential normalization. m(∈), so that when ∈ is given, the normalized results of all components can be regarded as a set of non-negative weights similar to probability distribution, and its normalization factor σ(∈) is used to regulate the relative difference between components. When σ(∈) takes a small value, the exponential function has a large or small effect on Ψ m (M(X),∈) will produce more significant amplification or weakening, making the distribution tend to extreme forms; if σ(∈) is relatively large, the gap between components can be smoothly compressed, making all Ψ m (M(X),∈) all occupy a certain proportion in the probability distribution. The advantage of this is that when the numerical values ​​of topological features in different dimensions are very different, a more reasonable weight distribution can be obtained through this mechanism, and other potential anomalies will not be covered up due to the excessive prominence of individual components. At the same time, the local entropy H(∈) defined on the basis of this set of probability distributions can reflect the overall uncertainty or chaos of Ψ(M(X),∈) at scale ∈. Specifically, if only a few components dominate at a certain ∈, and their normalized weights are significantly higher than other components, H(∈) will be low; on the contrary, if the sizes of all components are relatively close and there is no obvious dominant term, the entropy value will increase. For multidimensional time series anomaly detection in big data scenarios, this local entropy can precisely reflect the degree of heterogeneity of the internal structure of the system to a certain extent: if some topological features suddenly stand out, resulting in a rapid increase in the proportion of the corresponding components after normalization, the local entropy will decrease significantly, thus becoming a sign of anomaly; if all components show a consistent upward or downward trend, it means that there may be systematic disturbances or global changes in this period. Furthermore, by gradually changing the value of ∈ and m By dynamically tracking H(∈) and H(∈), the system can observe the fluctuation curve of local entropy at multiple scales, and then determine whether the anomaly is limited to a certain scale range, or whether it persists in the broad spectrum topological filtering process.

[0123] For industrial monitoring or IoT massive data, if at certain small scales, the Ψ m (M(X),∈) is extremely prominent, then the corresponding p m (∈) will be very biased towards these components, and the local entropy will also show rapid jumps near this scale; if this situation is still obvious under the subsequent larger ∈, it means that the structural deformation caused by the potential anomaly has passed through the filtering process of multiple scales and is unlikely to be isolated noise or short-term random disturbances. In the automated operation and maintenance scenario, the algorithm can calculate multiple groups of H(∈) for discrete values ​​of ∈, and then correspond them to the time axis to capture the entropy fluctuation characteristics generated in the sliding window at different times. The dimensionality reduction and manifold construction performed by the autoencoder in the previous steps provide these topological components Ψ m(M(X), ∈) lays a solid foundation for the acquisition of data, so that reasonable topological signatures can be calculated through unified spatial metrics in high-dimensional heterogeneous environments. Since the present invention uses exponential normalization in the calculation of local entropy, it can maintain a balanced consideration of different components in the face of extreme situations, thereby avoiding certain biases that occur in traditional entropy methods on multidimensional data. Such an entropy value also has a certain degree of interpretability: when H(∈) is low, it often means that the data distribution on the manifold presents a highly unbalanced pattern, and the topological features of some channels or sub-communities are extremely prominent, which may correspond to anomalies; when H(∈) is high, the system is in a relatively smooth or dispersed structural state, which does not necessarily mean anomalies, but is more likely to reflect the consistency or uniform distribution of the signal. If you want to accurately locate which dimensions or features have significantly contributed to the change in entropy, you can also directly observe p m The distribution of (∈) can be used to determine whether a major deviation has occurred at the instantaneous amplitude level, the continuous coherence level, or the wavelet packet energy level. On the other hand, the original intention of designing the normalization factor σ(∈) is to flexibly adjust the sensitivity of entropy calculation to numerical differences at different scales. By moderately relaxing or tightening σ(∈), the system can debug the response intensity when certain topological components deviate significantly in specific application scenarios, thereby making the present invention suitable for complex environments that require large-scale monitoring and often face small-probability severe anomalies. In industrial sites or large distributed networks, this type of multidimensional data accompanied by noise and interference at any time is particularly common. By dynamically adjusting σ(∈) and combining the aforementioned barcode information, it is often possible to quickly capture sudden failures while taking into account the identification of progressive hidden dangers.

[0124] Embodiment 9: In step 3, the abnormal probability is calculated according to the local entropy by the following formula:

[0125]

[0126] Among them, P anom is the abnormal probability.

[0127] Specifically, the present invention introduces a formula based on local entropy The function maps the entropy value to the interval between [0, 1], thus obtaining an abnormal probability that is easy to interpret and can be flexibly set. When the local entropy value H(∈) is small and lower than 0.5, it means that the data is strongly unbalanced or concentrated in the manifold space at this scale. It may also mean that the topological characteristics of some dimensions or channels are abnormally prominent, resulting in a decrease in the disorder of the overall structure of the system; for this period or this component, the abnormal probability P anomIt will increase and gradually approach 1, thus causing the system to pay attention. When H(∈) is at or above 0.5, it means that the entropy value of the multidimensional data feature is relatively balanced, and no one or several dimensions have extreme fluctuations in the topological structure, so the probability of abnormality in this area tends to remain at a relatively low level. In the big data scenario, such a function form provides a relatively smooth and segmented adjustable mapping. It will not produce drastic switching changes due to slight fluctuations in the entropy value, but can gradually adjust the abnormal probability according to the upper and lower deviations of the entropy value, thereby reducing false alarms caused by noise interference. At the same time, this also allows decision makers to directly observe whether the deviation of local entropy is sufficient to trigger the weighting mechanism at the reference point of 0.5, so that the system can make corresponding interventions or alarm prompts. For example, if the local entropy in multiple consecutive scale intervals is lower than 0.5 at a certain moment, and the corresponding abnormal probability exceeds a certain safety threshold, it can be combined with the manifold structure and topological barcode mentioned above to further confirm whether there is a systemic failure. Because the present invention emphasizes the combination of wavelet packet energy, instantaneous amplitude and instantaneous frequency, and continuous coherence into the autoencoder and manifold space, the local entropy calculated each time has integrated the contribution of each component; if some potential risks only appear in the short term or in a small dimension, then the local entropy may still be high at most scales, thereby keeping the probability of abnormality in a medium or low range to avoid overly sensitive alarms. When some large-scale, multi-dimensional outliers occur, a sharp drop in local entropy will be quickly converted into a high probability of abnormality, helping the system to accurately screen possible sudden failures or long-term hidden dangers in massive data streams. It is worth noting that in practical applications, coefficients such as -0.5 can be moderately adjusted because the definition and tolerance range of abnormalities are different in different industries or scenarios. If you need to be particularly sensitive to any low entropy situation in some high-risk process flows, you can increase the coefficient appropriately so that a slight drop in entropy will significantly increase the probability of abnormality; and for systems that require smooth operation but tolerate a small amount of violent fluctuations, you can reduce the coefficient to make the rise of abnormal probability smoother to reduce false alarms. The present invention also allows adaptive learning on the threshold of entropy: if it is found in the training phase or historical data that the system is actually in safe mode when the local entropy is generally in a certain interval, the offset or steepness of the mapping function can be automatically fine-tuned during online detection, thereby further improving the accuracy and flexibility of detection. In addition, this probability mapping based on the deformation of the Sigmoid function also has good numerical stability. For entropy values ​​in extreme cases, such as extremely low entropy or extremely high entropy, it can converge between 0 and 1 without causing numerical overflow problems. Moreover, the function presents a slow gain in the middle area, so that the normal fluctuations of local entropy will not easily trigger false alarms, and the probability of abnormality will only be pushed up when there are indeed significant signs of outliers.This corresponds to the abnormal features captured in the persistent coherence and wavelet packets and Hilbert instantaneous amplitude, allowing real systematic or significant anomalies to emerge quickly.

[0128] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A big data anomaly detection method based on time series multidimensional autoencoder, characterized in that: The method comprises: Step 1: Decompose the multidimensional time series data. First, use the Hilbert-Huang transform to extract the intrinsic oscillation mode of the multidimensional time series data. Then, use the wavelet packet transform to capture the local features at different scales and time periods. Adaptively extract the multi-level spatiotemporal characteristics from the multidimensional time series data and reconstruct them as the decomposition result. Step 2: Map the decomposition results into a low-dimensional and continuous manifold space; on the obtained manifold space, extract the structural information of the multidimensional time series data by calculating topological invariants such as continuous homology; Step 3: Based on the extracted structural information, calculate the local entropy of the multidimensional time series data at the set scale; based on the local entropy, calculate the anomaly probability.

2. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 1 is characterized in that: In step 1, assume that the multidimensional time series data is: Where M represents the dimension of multidimensional time series data; x m (t) is the m-th dimension of multidimensional time series data, m = 1, 2, ... M; T is the observation duration or the number of discrete sampling points; t is time; when the Hilbert-Huang transform is used to extract the intrinsic oscillation mode of multidimensional time series data, for each dimension x m (t) is processed as follows: x m (t) is decomposed into a set of intrinsic mode functions and a residual term: Among them, c m，k (t) is the kth intrinsic mode function of the mth dimension multidimensional time series data; K m is the total number of intrinsic mode functions of the m-th dimension multidimensional time series data; r m (t) is the residual term of the m-th dimension multidimensional time series data; for the k-th intrinsic mode function, the initial candidate intrinsic mode function is defined as: l is an integer subscript index; in the current candidate intrinsic mode function In the above example, we find all local maxima and local minima, and obtain the upper envelope by spline interpolation. With lower envelope The local average envelope is defined as: Update the candidate intrinsic mode function to: The standardized error is calculated as: When SD<∈, ∈ is the preset convergence threshold, it is considered The intrinsic mode function condition is satisfied, which is recorded as 3. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 2 is characterized in that: In step 1, for each intrinsic mode function c m，k (t) The Hilbert-Huang transform is calculated using the following formula: Where pv represents the Cauchy principal value integral; τ is the time integral variable; the corresponding analytical multidimensional time series data z is constructed by the following formula m，k (t): Where j is an imaginary unit, satisfying j 2 = -1; is the instantaneous amplitude; is the instantaneous phase; is the instantaneous frequency.

4. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 3 is characterized in that: In step 1, for each intrinsic mode function c obtained m，k (t) Wavelet packet transform is used to capture local features of different scales and time periods; let the mother wavelet be ψ(t), and construct the wavelet packet basis function corresponding to scale v and translation l: ψ v，l (t)=2 -v / 2 ψ(2 -v tl); Among them, v∈{v min , v min +1, ..., v max } is the scale; v min and v max Represent the minimum scale and the maximum scale respectively; the intrinsic mode function c m，k (t) in the basis ψ v，l (t) and calculate the coefficient as local features.

5. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 4 is characterized in that: In step 1, for the m-th dimension of multidimensional time series data, the reconstruction expression is: Among them, Λ v Represents the set of all valid translation indices under scale v; α m，k (v, l) is the reconstruction coefficient, which is calculated by taking into account the wavelet packet coefficient W m，k (v, l) and the instantaneous amplitude A extracted by the Hilbert-Huang transform m，k (t) and frequency ω m，k (t), defined as: α m，k (v,l)=0.3·W m，k (v,l)+0.4·A m，k (t)+0.3·ω m，k (t); is the decomposition result of the m-th dimension multidimensional time series data. The decomposition result of the multidimensional time series data is:

6. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 5, characterized in that: In step 2, the decomposition result is mapped to a low-dimensional and continuous manifold space by the following formula: in, The obtained manifold space is a smooth manifold; is the Kronecker product operation; ||·|| HS is the Hilbert-Schmidt norm; τ is the time integral variable; R(τ) is the Riemann curvature tensor.

7. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 6 is characterized in that: In step 2, on the obtained manifold space, the structural information of the multidimensional time series data is extracted by calculating topological invariants such as continuous homology through the following formula: Among them, VR(M(X),∈) represents the The constructed Vietoris–Rips complex; H m (VR(M(X),∈)) is the m-th dimensional homology group under scale ∈; γ m，i is the i-th homology class in the m-th dimension; b m，i With d m，i They are γ m，i The birth scale and death scale of each barcode pair (b m，i d m，i ) is converted into numerical features; inf{·} is the lower bound operation; inf{∈:γ m，i vanishes} means that for a given homology class γ m，i , in all scales ∈ during the filtering process, such that γ m，i The infimum of those ∈ that disappear; For a given homology class γ m，i , find the minimum value of the scale parameter ∈ in the filtering process so that γ m，i First appeared in the m-th dimensional homology group H m (VR(M(X),∈)); is the structural information of the m-th dimension of multidimensional time series data under scale ∈.

8. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 7, characterized in that: In step 3, the structural information of the multidimensional time series data is assumed to be: Ψ(M(X),∈)=[Ψ1(M(X),∈),Ψ2(M(X),∈),...,Ψ M (M(X),∈)]; Exponential normalization is applied to each component to define the normalized probability distribution: m=1,2,...,M;c=1,2,...,M;σ(∈) is the normalization factor under scale ∈, which is the set value; using the normalized probability distribution p m (∈) Calculate the local entropy H(∈) at each scale ∈:

9. The big data anomaly detection method based on time series multidimensional autoencoder according to claim 8, characterized in that: In step 3, the anomaly probability is calculated based on the local entropy using the following formula: Among them, P anom is the abnormal probability.

Images