Data resource integration system based on cloud computing

Abstract

The invention discloses a data resource integration system based on cloud computing, and relates to the technical field of data resource processing, cloud platform data resources are preprocessed through a sequence generation module, timestamps are extracted, standardized time sequences are generated, and t-SNE is used for dimension reduction; the network construction module constructs a similarity complex network according to the sequence after dimension reduction, calculates the similarity of data points and constructs a simplified complex; the persistent calculation module calculates topological features of the simplified complex at different scales through a persistent coherence algorithm to generate a persistent coherence graph; the drift identification module compares the persistent homology graphs in different time periods, calculates a Wasserstein distance and identifies data drift; and the resource integration module classifies and screens data resources according to the drifting result, retains valid data, eliminates invalid data and realizes dynamic integration. And comparing and analyzing the persistent homograph, intelligently classifying and screening data resources based on a drift identification result, automatically eliminating invalid or noise data, and retaining data resources valuable for analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of data resource processing, and in particular to a data resource integration system based on cloud computing. Background Art

[0002] With the rapid development of big data and cloud computing, various data resources have shown diversified and dynamic characteristics on different cloud computing platforms. These data resources face many challenges in practical applications due to their diverse sources, complex formats, and variable spatiotemporal characteristics. For example, existing data integration technologies usually only process data in a specific format and lack effective means to integrate multi-source heterogeneous data. In addition, existing methods mostly rely on direct processing of high-dimensional data and fail to effectively use dimensionality reduction technology to reduce data noise and improve analysis efficiency, resulting in high computational complexity and low data utilization.

[0003] Existing network construction methods are mainly based on single-dimensional data association, which cannot fully utilize the association of multi-dimensional time series data and is difficult to maintain the stability and consistency of the network structure in a dynamically changing data environment. At the same time, the application of persistent computing technology is relatively limited, and the ability to maintain the characteristics of complex data forms is insufficient, and it is unable to effectively identify the long-term change trend of data resources.

[0004] In terms of data drift identification, most current methods are limited to data analysis of a single time period, lacking comparative analysis of persistence characteristics between different time periods, making it difficult to accurately identify the drift pattern of data resources. In addition, existing data resource classification and screening methods are mostly based on static rules, which are difficult to adapt to dynamic changes in data and easily lead to the accumulation of redundant or invalid data, affecting the accuracy of data analysis results. Summary of the invention

[0005] Based on the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide a data resource integration system based on cloud computing to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solutions: A data resource integration system based on cloud computing, comprising:

[0007] A sequence generation module is used to preprocess all data resources on the cloud computing platform, extract timestamps, generate standardized time series inputs, and perform dimensionality reduction operations on the time series inputs according to the t-SNE dimensionality reduction algorithm;

[0008] The network construction module is used to construct a similarity complex network based on the time series input after dimensionality reduction, calculate the similarity between data points, construct a topological network, and construct a simplified complex based on the similarity between the topological network and the data points;

[0009] A persistence calculation module is used to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm to generate a persistent homology graph;

[0010] The drift identification module is used to compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution;

[0011] The resource integration module is used to classify data resources according to the drift identification results of data resource distribution, screen out valid data resources, eliminate invalid data resources, and complete the dynamic integration of data resources.

[0012] The present invention is further configured such that the sequence generation module includes a preprocessing unit, a sequence generation unit and a dimensionality reduction unit; the preprocessing unit is used to perform noise reduction, normalization and standardization processing on all data resources on the cloud computing platform, and extract timestamps; the sequence generation unit is used to extract change characteristics of data resources, and generate standardized time series inputs according to the data resources, change characteristics and timestamps; the dimensionality reduction unit is used to perform dimensionality reduction operations on the time series inputs according to the t-SNE dimensionality reduction algorithm, map the high-dimensional time series inputs to a low-dimensional space, and retain local structures and global topological features.

[0013] The present invention is further configured to extract the change characteristics of the data resources and generate a standardized time series input according to the data resources, the change characteristics and the timestamp, including:

[0014] Decomposing the time series of the data resource into a trend part, a seasonal part and a residual part by using a local weighted regression algorithm, and setting the seasonal part as a seasonal variation feature;

[0015] Nonlinearly summing and normalizing the deviations of data points in a preset window from the central trend, calculating the deviation volatility, and setting the deviation volatility as a fluctuation change feature;

[0016] Obtain an autocorrelation coefficient by calculating the covariance between the current value and the lagged value of the data resource, and set the autocorrelation coefficient as an autocorrelation feature;

[0017] Through feature concatenation, the standardized data, change features, and timestamps in the data resources are combined into a time series input matrix as the time series input.

[0018] The present invention is further configured such that the calculation logic of the similarity between data points is: Among them, De ij is the similarity between the i-th data point and the j-th data point, Y i and Y jare the vector representations of the data points after dimensionality reduction, ||·|| is the symbol for calculating the Euclidean distance, and t i and t j are the timestamps corresponding to the i-th data point and the j-th data point, respectively. TimeFactor(t i ,t j ) is the time correction factor, and the calculation logic is: TimeFactor(t i ,t j )=1+λ·|t i -t j | ρ , λ is the time influence factor, ρ is the time adjustment factor, which is used to adjust the influence of time difference on similarity, β is the adjustment parameter, which is used to control the nonlinear amplification effect of distance, and γ is the scale factor, which is used to control the attenuation speed of distance;

[0019] A topological network is constructed by taking data points as nodes of the topological network and the similarities between data points as edges between nodes;

[0020] In constructing a simplified complex shape according to the similarity between the topological network and the data points, the construction logic of the simplified complex shape is: Among them, R(∈) is a simplified complex, which contains a set of node subsets; σ is a simplex of the topological network, which represents the connection relationship of multiple nodes; V is the node set of the nodes of the topological network, and v i and v j are the i-th and j-th nodes of the topological network, ∈ is the similarity threshold, which is used to eliminate insignificant similarity relationships and simplify the complex.

[0021] The present invention is further configured to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm to generate a persistent homology graph, including:

[0022] The scale of the preset filtering parameter is gradually increased according to the preset increment function to construct a complex sequence;

[0023] Performing persistent homology calculations on the complex sequence to identify the appearance and disappearance of different topological features;

[0024] According to the results of persistent homology calculation, the life cycle of topological features is plotted as a persistent homology graph.

[0025] The present invention is further configured such that the construction logic of the complex sequence is: α ={R(∈)|∈∈{∈ 0 ,∈ 1 ,…,∈ n},∈ i+1 =∈ i +Δ(α)}, where F αis a complex sequence with filtering process, that is, a multi-level complex constructed by gradually increasing the scale parameter ∈, α is the filtering adjustment parameter, R(∈) is the simplified complex, the complex constructed under different α, n is the number of complexes in the sequence, Δ(α) is the increment function, and the calculation logic is: Among them, δ is a nonlinear adjustment parameter, which is used to ensure that the scale increment is dynamically adjusted under different complexities of the complex;

[0026] The persistent homology calculation is performed on the complex sequence, and the calculation logic is: PH(F α )={(b k ,d k )|k=1,2,…,m}, where PH(F α ) is the complex sequence F α The persistent homology result contains a series of lifetime pairs of topological features (b k ,d k ), b k and d k They are respectively represented as the appearance and disappearance scales of the kth topological feature;

[0027] The drawing logic of the persistent homology graph is: Among them, PD is a persistent homology graph, which maps the lifetime of topological features to (b k ,d k ) is graphically represented, with the horizontal axis representing the birth scale and the vertical axis representing the death scale, to observe the life span of topological characteristics.

[0028] The present invention is further configured to compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution, including:

[0029] Calculate the Wasserstein distance between the persistent homology graphs in adjacent time periods and generate the distance value between the persistent homology graphs;

[0030] Based on the distance values ​​between the persistent homology graphs, the distance value change sequence is calculated, and the drift significance score is calculated according to the distance value change sequence according to the preset drift significance calculation logic.

[0031] The present invention is further configured such that the calculation logic of the distance value between the persistent homology graphs is: in, t i Time and t j Wasserstein distance between time-persistent homology graphs, is a matching solution set, indicating that The point in All possible combinations of point matches in , ∥xy∥ is the Euclidean distance between two points, ε is a nonlinear amplification parameter used to adjust the sensitivity to distance, To match the cost function, the calculation logic is: Among them, h(x) is the topological level function, which is used to quantify the scale position of the data point x in the persistent homology graph, κ is the weight coefficient, and θ is the time correlation adjustment parameter, which is used to adjust the topological level difference of the matching points;

[0032] The calculation logic of the distance value change sequence is: Among them, Δ t is a distance value change sequence, which contains the distance values ​​between the persistent homology graphs of all adjacent time periods;

[0033] The default drift significance calculation logic is: Where S is the drift significance score, T is the number of time periods, which means the total number of time periods compared, that is, the total number of persistent homology graph comparison calculations, Φ(·) is the cumulative distribution function, which is used to measure the significance level of Wasserstein distance, and μ w is the expected value, used as the benchmark value for distance significance test, σ w is a scale parameter used to measure the volatility of distance.

[0034] The present invention is further configured to classify data resources according to the drift identification result of data resource distribution, specifically including:

[0035] When the drift significance score is greater than the preset drift significance threshold, it means that the data resource drifts significantly. The latest data resource is retained by dichotomy according to the time period, and the drift significance score is recalculated until the drift significance score is less than or equal to the preset drift significance threshold.

[0036] When the drift significance score is less than or equal to the preset drift significance threshold, it indicates that the data resource drift is not significant, and the data resource is set as a valid resource.

[0037] The present invention is further configured to include a dimensionality reduction quality evaluation unit, which is used to evaluate the time series input after dimensionality reduction. When the quality metric is greater than a preset quality threshold, it means that the time series input after dimensionality reduction is distorted and cannot represent data resources, and the dimensionality is re-reduced to a quality metric less than or equal to the preset quality threshold. The calculation logic of the dimensionality reduction quality evaluation is: Q = ∑ i,j (p ij -q ij ) 2 , Q is the quality measure, p ij is the similarity probability between data point i and data point j in the time series input. The calculation logic is: ∥Xscaled,i -X scaled,j ∥ is the Euclidean distance between data point i and data point j in the time series input, σ i is the scale parameter used to calculate the similarity probability, adjusting the width of the similarity distribution around point i, q ij is the similarity probability between data point i and data point j in the time series input after dimensionality reduction. The calculation logic is: Among them, ∥Y i -Y j ∥ is the Euclidean distance between data point i and data point j in the time series input after dimensionality reduction.

[0038] The present invention provides a data resource integration system based on cloud computing, comprising a sequence generation module, which is used to pre-process all data resources on a cloud computing platform, extract timestamps, generate standardized time series inputs, and perform dimension reduction operations on the time series inputs according to a t-SNE dimension reduction algorithm; a network construction module, which is used to construct a similarity complex network according to the time series input after dimension reduction, calculate the similarity between data points, construct a topological network, and construct a simplified complex according to the similarity between the topological network and the data points; a persistent calculation module, which is used to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm, and generate a persistent homology graph; a drift identification module, which is used to compare persistent homology graphs of different time periods, calculate the Wasserstein distance between persistent homology graphs of each time period, and identify the drift of data resource distribution; a resource integration module, which is used to classify data resources according to the drift identification results of data resource distribution, screen out effective data resources, eliminate invalid data resources, and complete the dynamic integration of data resources, and the beneficial effects generated include:

[0039] 1. Automatic construction and feature mining of complex networks: The present invention automatically constructs complex networks of multidimensional time series data through network construction modules, and visualizes the network structure based on the similarity and correlation between nodes. This function can reveal the potential correlation between data, making the data structuring and analysis process more intuitive and efficient, and solving the problem of difficulty in constructing multidimensional correlation networks in the prior art;

[0040] 2. Persistent feature calculation and mutation identification: The persistent calculation module can analyze the morphological changes of data in different time periods, identify and quantify the persistent features of data resources, and can not only reveal the long-term trend of data changes over time, but also intuitively display the evolution process of data features through persistent homology diagrams, helping users to deeply understand the persistent characteristics of data;

[0041] 3. Accurate data drift identification: The drift identification module of the present invention can compare and analyze the persistent homology graphs of different time periods, and identify the drift trend of data resource distribution by calculating the Wasserstein distance between each time period. This function can timely discover the change pattern and mutation point of data resources, help to dynamically adjust the data analysis strategy, and improve the flexibility and accuracy of data management;

[0042] 4. Intelligent data resource integration and screening: The present invention uses a resource integration module to intelligently classify and screen data resources based on drift identification results, automatically remove invalid or noisy data, and retain data resources that are valuable for analysis. This method overcomes the limitations of traditional rule-based screening, can adapt to the dynamic changes of data resources, and ensure the efficiency and accuracy of data analysis results.

[0043] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0045] Figure 1 The present invention is a schematic structural diagram of a data resource integration system based on cloud computing, showing an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will describe the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention, not for limiting the scope of protection of the present invention.

[0047] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0048] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0049] A data resource integration system based on cloud computing, such as Figure 1 As shown, including:

[0050] A sequence generation module is used to preprocess all data resources on the cloud computing platform, extract timestamps, generate standardized time series inputs, and perform dimensionality reduction operations on the time series inputs according to the t-SNE dimensionality reduction algorithm;

[0051] The network construction module is used to construct a similarity complex network based on the time series input after dimensionality reduction, calculate the similarity between data points, construct a topological network, and construct a simplified complex based on the similarity between the topological network and the data points;

[0052] A persistence calculation module is used to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm to generate a persistent homology graph;

[0053] The drift identification module is used to compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution;

[0054] The resource integration module is used to classify data resources according to the drift identification results of data resource distribution, screen out valid data resources, eliminate invalid data resources, and complete the dynamic integration of data resources.

[0055] Specifically, the main task of the sequence generation module is to preprocess all data resources on the cloud computing platform, extract timestamps, and generate standardized time series inputs. After that, the time series inputs are subjected to dimensionality reduction processing through the t-SNE (t-Distributed Stochastic Neighbor Embedding) dimensionality reduction algorithm so as to better reflect the structure and dynamic changes of the data; the present invention is further configured such that the sequence generation module includes a preprocessing unit, a sequence generation unit and a dimensionality reduction unit; the preprocessing unit is used to perform denoising, normalization and standardization processing on all data resources on the cloud computing platform, and extract timestamps; the sequence generation unit is used to extract the change characteristics of the data resources, and generate standardized time series inputs according to the data resources, the change characteristics and the timestamps; the dimensionality reduction unit is used to perform dimensionality reduction operations on the time series inputs according to the t-SNE dimensionality reduction algorithm, and map the high-dimensional time series inputs to the low-dimensional space, retaining the local structure and the global topological characteristics. t-SNE (t-Distributed Stochastic Neighbor Embedding) is a nonlinear dimensionality reduction algorithm that can embed high-dimensional data into a low-dimensional space while retaining the local and global structural relationships between data points. It calculates the similarity probability of data points in the high-dimensional space and finds the best embedding in the low-dimensional space, so that the original similarity relationship is retained to the greatest extent. It is an existing technology and will not be described in detail here.

[0056] Specifically, preprocessing includes noise reduction, normalization and standardization; noise reduction: use filtering technology (such as low-pass filter) or statistical methods (such as median filtering) to reduce noise on the data, remove unnecessary high-frequency noise, and enhance the signal quality of the data; normalization and standardization: normalize the data to scale data with different characteristics to a uniform range (such as [0,1]), and eliminate the influence of dimension through standardization (making the data conform to the standard normal distribution), thereby enhancing the comparability of different data features; preprocessing operations ensure that the data remains consistent in the subsequent processing process, reducing analysis errors caused by noise and inconsistent dimensions.

[0057] The present invention is further configured to extract the change characteristics of the data resources and generate a standardized time series input according to the data resources, the change characteristics and the timestamp, including:

[0058] The local weighted regression algorithm is used to decompose the time series of data resources into trend, seasonal and residual parts, and the seasonal part is set as the seasonal variation feature; specifically, the seasonal component is used to capture the periodic repetitive pattern in the time series, and the seasonal decomposition is to decompose the time series into trend, seasonal and residual parts: the decomposition logic is: t =T t +St +R t ,y t is the original time series data point value, T t is the trend component, which indicates the long-term trend of data, S t is the seasonal component, capturing the periodic changes in the data, R t is the residual component, representing the random changes not explained by the trend and seasonality; the seasonal component extraction steps are: smoothing the original time series to eliminate short-term fluctuations, thereby extracting the trend component T t , subtract the trend component T from the original series t , and then extract S by period averaging or weighted method t , calculate the residual R t =y t -T t -S t , used to detect anomalies and noise.

[0059] The deviations of the data points in the preset window from the central trend are nonlinearly summed and normalized, the deviation volatility is calculated, and the deviation volatility is set as the fluctuation change feature; specifically, the deviation volatility is used to measure the fluctuation amplitude of the time series data relative to the central trend, and the calculation logic is: Med t =Median(x t-n ,…,x t ), where DV t is the deviation volatility at time t, x i is the sequence data point value within the window, Med t The central tendency of the data in the window is calculated using the median to reduce the impact of outliers. ω1 is a nonlinear adjustment parameter used to enhance the sensitivity to drastic changes. It is usually set to 1 or 2 to adjust the amplification effect of deviations.

[0060] The covariance between the current value and the lagged value of the data resource is calculated to obtain an autocorrelation coefficient, and the autocorrelation coefficient is set as an autocorrelation feature; specifically, the calculation logic of the covariance between the current value and the lagged value is: Where ACF(k) is the covariance between the current value and the lagged value, x t is the data point value in the time series, μ is the average level of the series, which can be replaced by a smoothing method, and k is the number of lags, which is between 1 and one cycle length.

[0061] Through feature splicing, the standardized data, change features and timestamps in the data resources are combined into a time series input matrix as the time series input; feature splicing is to connect multiple feature vectors along a certain dimension to form a new feature vector. The splicing method can be horizontal splicing (increasing on the feature dimension) or vertical splicing (increasing on the sample dimension), which is the splicing of the feature dimension in the present invention; the core of the above logic is to extract the key change features of the data resources through the decomposition, volatility analysis and autocorrelation detection of the time series data, and standardize them to generate a stable and predictive time series input, including the extraction of trend components, seasonal components and residual components, the calculation of volatility, and the evaluation of autocorrelation functions, and finally integrate these features into a standardized time series for subsequent analysis and processing.

[0062] The present invention is further configured such that the calculation logic of the similarity between data points is: Among them, De ij is the similarity between the i-th data point and the j-th data point, Y i and Y j are the vector representations of the data points after dimensionality reduction, ||·|| is the symbol for calculating the Euclidean distance, and t i and t j are the timestamps corresponding to the i-th data point and the j-th data point, respectively. TimeFactor(t i ,t j ) is the time correction factor, and the calculation logic is: TimeFactor(t i ,t j )=1+λ·|t i -t j | ρ , λ is the time influence factor, ranging from 0.1 to 1, which is used to control the influence of time factors on similarity, ρ is the time adjustment factor, which is used to adjust the influence of time difference on similarity, with a value of 1 or 2, β is the adjustment parameter, which is used to control the nonlinear amplification effect of distance, ranging from 1 to 3, which is used to adjust the amplification effect of distance in similarity calculation. The larger the value, the more significant the difference in distance, γ is the scale factor, which is used to control the attenuation speed of distance, ranging from 0.5 to 2, to ensure that the similarity value does not deviate too much; the above calculation logic combines spatial distance and time difference to ensure that the calculated similarity can reflect the geometric relationship of data points and capture the dynamic changes of time factors;

[0063] A topological network is constructed with data points as nodes of the topological network and the similarities between data points as edges between nodes. In the topological network, node pairs with larger similarity values ​​are regarded as closely connected nodes, thus forming a stronger topological connection in the network.

[0064] In constructing a simplified complex shape according to the similarity between the topological network and the data points, the construction logic of the simplified complex shape is: Among them, R(∈) is a simplified complex, which contains a set of node subsets; σ is a simplex of the topological network, which represents the connection relationship of multiple nodes; V is the node set of the nodes of the topological network, and v i and v j are the ith and jth nodes of the topological network, ∈ is the similarity threshold, ranging from 0.1 to 0.5, which is used to eliminate insignificant connection relationships, retain only key topological structures, and simplify the complex; by setting the threshold ∈, only node pairs with high similarity are retained, and a simplified complex is constructed to capture important geometric and topological relationships in the data. Specifically, the core task of the above logic is to construct a similarity complex network by calculating the similarity between data points, and to construct a simplified complex based on the similarity. The specific method is to first use measurement methods such as Euclidean distance combined with time factors to calculate the similarity, and then filter the connection relationships between data points according to the similarity value, and finally generate a simplified complex, which can accurately describe the geometric and topological relationships between data points and provide a basis for subsequent topological analysis.

[0065] The present invention is further configured to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm to generate a persistent homology graph, including:

[0066] The scale of the preset filtering parameter is gradually increased according to the preset increment function to construct a complex sequence;

[0067] Performing persistent homology calculations on the complex sequence to identify the appearance and disappearance of different topological features;

[0068] According to the results of persistent homology calculation, the life cycle of topological features is plotted as a persistent homology graph.

[0069] The present invention is further configured such that the construction logic of the complex sequence is: α ={R(∈)|∈∈{∈ 0 ,∈ 1 ,…,∈ n},∈ i+1 =∈ i +Δ(α)}, where F α is a complex sequence with a filtering process, that is, a multi-level complex constructed by gradually increasing the scale parameter ∈, α is a filtering adjustment parameter, ranging from 0.1 to 1, which adjusts the scale increment during complex construction to ensure that the scale increment can reasonably capture the multi-level characteristics of the data, R(∈) is a simplified complex, a complex constructed under different α, n is the number of complexes in the sequence, Δ(α) is the increment function, and the calculation logic is: Among them, δ is a nonlinear adjustment parameter, which is used to ensure that the scale increment is dynamically adjusted under different complexities of the complex; the complex sequence is a set of complexes at different scales. By adjusting the filtering parameters, the complexity of the complex is gradually increased, and the change process of the topological structure is gradually captured. By increasing the scale and gradually increasing the complexity of the complex, the changing characteristics of the data topological structure at multiple scales can be captured, laying the foundation for persistent homology calculation;

[0070] The persistent homology calculation is performed on the complex sequence, and the calculation logic is: PH(F α )={(b k ,d k )|k=1,2,…,m}, where PH(F α ) is the complex sequence F α The persistent homology result contains a series of lifetime pairs of topological features (b k ,d k ), b k and d k They are respectively represented as the scale of appearance and disappearance of the kth topological feature; by calculating persistent homology, stable and short-lived topological features in the data are identified, and the life cycle of these features at different scales is quantified;

[0071] The drawing logic of the persistent homology graph is: Among them, PD is a persistent homology graph, which maps the lifetime of topological features to (b k ,d k ) is graphically represented, with the horizontal axis being the birth scale and the vertical axis being the death scale, to observe the life span of topological features; the persistence of topological features is graphically displayed, and the changes in data structure at different scales are intuitively presented, which is convenient for analyzing and interpreting the topological morphology of the data. Specifically, the core of the above-mentioned calculation logic is to analyze the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm, and generate a persistent homology graph. The specific steps include constructing a complex sequence, performing persistent homology calculation on the complex sequence, and drawing a persistent homology graph. By incrementally increasing the scale, the emergence and disappearance process of different topological features is gradually captured, and the life cycle of these features is visualized, providing a reliable basis for the topological analysis of the data.

[0072] The present invention is further configured to compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution, including:

[0073] The Wasserstein distance between the persistent homology graphs in adjacent time periods is calculated to generate the distance value between the persistent homology graphs; the Wasserstein distance is used to measure the difference between two persistent homology graphs, and the similarity is calculated by minimizing the matching cost between the two graphs; the present invention is further configured such that the calculation logic of the distance value between the persistent homology graphs is: in, t i Time and t j Wasserstein distance between time-persistent homology graphs, is a matching solution set, indicating that The point in All possible combinations of point matches in , ∥xy∥ is the Euclidean distance between two points, ε is a nonlinear amplification parameter used to adjust the sensitivity to distance, and its value range is 1 to 3. To match the cost function, the calculation logic is: Among them, h(x) is the topological hierarchy function, which is used to quantify the scale position of the data point x in the persistent homology graph, κ is the weight coefficient, which ranges from 0.5 to 2 and is used to adjust the influence of the hierarchy difference in the matching cost, and θ is the time correlation adjustment parameter, which is used to adjust the topological hierarchy difference of the matching point, and ranges from 1 to 2; matching cost function It is used to measure the topological differences between matching point pairs in the persistent homology graph and optimize the matching cost calculation by considering the nonlinear differences in the topological level.

[0074] Based on the distance values ​​between the persistent homology graphs, the distance value change sequence is calculated, and the drift significance score is calculated according to the preset drift significance calculation logic based on the distance value change sequence. The calculation logic of the distance value change sequence is: Among them, Δ t It is a distance value change sequence, including the distance values ​​between all the persistent homology graphs in adjacent time periods; the preset drift significance calculation logic is: Where S is the drift significance score, T is the number of time periods, which means the total number of time periods compared, that is, the total number of persistent homology graph comparison calculations, Φ(·) is the cumulative distribution function, which is used to measure the significance level of Wasserstein distance, and μ w is the expected value, which is used as the benchmark value for distance significance test. It is determined based on historical data or theoretical distribution and is used as the benchmark for significance detection. wis a scale parameter, which is usually obtained by fitting the distribution of historical Wasserstein distances and is used to measure the volatility of distances. Specifically, the core of the above calculation logic is to identify the drift of data resource distribution and evaluate the significance of drift by calculating the Wasserstein distance between persistent homology graphs of different time periods. First, the difference between persistent homology graphs of adjacent time periods is calculated by the Wasserstein distance to form a distance value sequence. The drift significance score is calculated based on the distance value sequence to judge the significance of data distribution changes, which can quantify the degree of change of data topological features in the time dimension.

[0075] The present invention is further configured to classify data resources according to the drift identification result of data resource distribution, specifically including:

[0076] When the drift significance score is greater than the preset drift significance threshold, it indicates that the data resource drifts significantly, and the latest data resource is retained by dichotomy according to the time period, and the drift significance score is recalculated until the drift significance score is less than or equal to the preset drift significance threshold; specifically, when the drift significance score is greater than the preset drift significance threshold, it indicates that the drift significance of the data resource exceeds the preset range, indicating that the data resource has significant drift and needs to be updated, and the latest data resource is retained by dichotomy according to the time period, and the drift significance score is recalculated to ensure the validity of the updated data;

[0077] When the drift significance score is less than or equal to the preset drift significance threshold, it indicates that the data resource drift is not significant, and the data resource is set as a valid resource; when the drift significance score is less than or equal to the preset drift significance threshold, it indicates that the drift significance of the data resource does not exceed the preset range, and the drift of the data resource is considered to be insignificant, and the data resource is set as a valid resource, retaining the existing data without additional update processing; through the significance score, data resources with significant changes can be quickly screened out, providing an accurate basis for the update and integration of data resources, and can flexibly adjust the update strategy of data resources to ensure the real-time and accuracy of data during use.

[0078] The present invention is further configured to include a dimensionality reduction quality evaluation unit, which is used to evaluate the time series input after dimensionality reduction. When the quality metric is greater than a preset quality threshold, it means that the time series input after dimensionality reduction is distorted and cannot represent data resources, and the dimensionality is re-reduced to a quality metric less than or equal to the preset quality threshold. The calculation logic of the dimensionality reduction quality evaluation is: Q = ∑ i,j (p ij -q ij ) 2 , Q is the quality measure, p ijis the similarity probability between data point i and data point j in the time series input. The calculation logic is: ∥X scaled,i -X scaled,j ∥ is the Euclidean distance between data point i and data point j in the time series input, σ i is the scale parameter used to calculate the similarity probability, which adjusts the width of the similarity distribution around point i, with a value range of 0.1 to 2, q ij is the similarity probability between data point i and data point j in the time series input after dimensionality reduction. The calculation logic is: Among them, ∥Y i -Y j ∥ is the Euclidean distance between data point i and data point j in the time series input after dimensionality reduction; the quality metric Q is used to evaluate whether the data after dimensionality reduction reasonably retains the similarity relationship of the original high-dimensional space. By comparing the similarity probability between point pairs in high-dimensional and low-dimensional spaces, the quality metric value is calculated to evaluate the dimensionality reduction effect. A larger Q value means that the similarity between the low-dimensional data points after dimensionality reduction is significantly different from the similarity in the original high-dimensional space. At this time, the dimensionality reduction algorithm fails to accurately maintain the local and global structural characteristics of the data, which may lead to distortion in subsequent data analysis and processing; specifically, the core of the above calculation logic is to perform quality assessment on the time series input after dimensionality reduction to ensure that the dimensionality reduction result can retain the core structure and similarity relationship of the data. By calculating the dimensionality reduction quality metric, it is determined whether the data after dimensionality reduction can be further processed and integrated. When the dimensionality reduction quality metric exceeds the preset threshold, it means that the dimensionality reduction effect is not ideal, and the dimensionality reduction strategy needs to be adjusted or the data needs to be reprocessed.

[0079] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0080] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0081] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0082] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0083] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0084] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0085] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0088] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data resource integration system based on cloud computing, characterized in that: include: A sequence generation module is used to preprocess all data resources on the cloud computing platform, extract timestamps, generate standardized time series inputs, and perform dimensionality reduction operations on the time series inputs according to the t-SNE dimensionality reduction algorithm; The network construction module is used to construct a similarity complex network based on the time series input after dimensionality reduction, calculate the similarity between data points, construct a topological network, and construct a simplified complex based on the similarity between the topological network and the data points; A persistence calculation module is used to calculate the persistence of topological features of simplified complexes at different scales through a persistent homology calculation algorithm to generate a persistent homology graph; The drift identification module is used to compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution; The resource integration module is used to classify data resources according to the drift identification results of data resource distribution, screen out valid data resources, eliminate invalid data resources, and complete the dynamic integration of data resources.

2. According to the cloud computing-based data resource integration system of claim 1, it is characterized in that: The sequence generation module includes a preprocessing unit, a sequence generation unit and a dimension reduction unit; the preprocessing unit is used to perform noise reduction, normalization and standardization processing on all data resources on the cloud computing platform and extract timestamps; The sequence generation unit is used to extract the change characteristics of data resources and generate a standardized time series input according to the data resources, the change characteristics and the timestamp; the dimension reduction unit is used to perform a dimensionality reduction operation on the time series input according to the t-SNE dimensionality reduction algorithm, map the high-dimensional time series input to a low-dimensional space, and retain the local structure and global topological characteristics.

3. The data resource integration system based on cloud computing according to claim 2, characterized in that: Extract the change characteristics of data resources and generate standardized time series input based on data resources, change characteristics and timestamps, including: Decomposing the time series of the data resource into a trend part, a seasonal part and a residual part by using a local weighted regression algorithm, and setting the seasonal part as a seasonal variation feature; Nonlinearly summing and normalizing the deviations of data points in a preset window from the central trend, calculating the deviation volatility, and setting the deviation volatility as a fluctuation change feature; Obtain an autocorrelation coefficient by calculating the covariance between the current value and the lagged value of the data resource, and set the autocorrelation coefficient as an autocorrelation feature; Through feature concatenation, the standardized data, change features, and timestamps in the data resources are combined into a time series input matrix as the time series input.

4. The data resource integration system based on cloud computing according to claim 1, characterized in that: The calculation logic of the similarity between data points is: Among them, De ij is the similarity between the i-th data point and the j-th data point, Y i and Y j are the vector representations of the data points after dimensionality reduction, ||·|| is the symbol for calculating the Euclidean distance, and t i and t j are the timestamps corresponding to the i-th data point and the j-th data point, respectively. TimeFactor(t i ,t j ) is the time correction factor, and the calculation logic is: TimeFactor(t i ,t j )=1+λ·|t i -t j | ρ , λ is the time influence factor, ρ is the time adjustment factor, which is used to adjust the influence of time difference on similarity, β is the adjustment parameter, which is used to control the nonlinear amplification effect of distance, and γ is the scale factor, which is used to control the attenuation speed of distance; A topological network is constructed by taking data points as nodes of the topological network and the similarities between data points as edges between nodes; In constructing a simplified complex shape according to the similarity between the topological network and the data points, the construction logic of the simplified complex shape is: Among them, R(∈) is a simplified complex, which contains a set of node subsets; σ is a simplex of the topological network, which represents the connection relationship of multiple nodes; V is the node set of the nodes of the topological network, and v i and v j are the i-th and j-th nodes of the topological network, ∈ is the similarity threshold, which is used to eliminate insignificant similarity relationships and simplify the complex.

5. The data resource integration system based on cloud computing according to claim 1, characterized in that: The persistence of topological features of simplified complexes at different scales is calculated by persistent homology calculation algorithm to generate persistent homology graphs, including: The scale of the preset filtering parameter is gradually increased according to the preset increment function to construct a complex sequence; Performing persistent homology calculations on the complex sequence to identify the appearance and disappearance of different topological features; According to the results of persistent homology calculation, the life cycle of topological features is plotted as a persistent homology graph.

6. The data resource integration system based on cloud computing according to claim 5, characterized in that: The construction logic of the complex sequence is: α ={R(∈)∣∈∈{∈0,∈1,…,∈ n },∈ i+1 =∈ i +Δ(α)}, where F α is a complex sequence with filtering process, that is, a multi-level complex constructed by gradually increasing the scale parameter ∈, α is the filtering adjustment parameter, R(∈) is the simplified complex, the complex constructed under different α, n is the number of complexes in the sequence, Δ(α) is the increment function, and the calculation logic is: Among them, δ is a nonlinear adjustment parameter, which is used to ensure that the scale increment is dynamically adjusted under different complexities of the complex; The persistent homology calculation is performed on the complex sequence, and the calculation logic is: PH(F α )={(b k ,d k )|k=1,2,…,m}, where PH(F α ) is the complex sequence F α The persistent homology result contains a series of lifetime pairs of topological features (b k ,d k ), b k and d k They are respectively represented as the appearance and disappearance scales of the kth topological feature; The drawing logic of the persistent homology graph is: Among them, PD is a persistent homology graph, which maps the lifetime of topological features to (b k ,d k ) is graphically represented, with the horizontal axis representing the birth scale and the vertical axis representing the death scale, to observe the life span of topological characteristics.

7. The data resource integration system based on cloud computing according to claim 1, characterized in that: Compare the persistent homology graphs of different time periods, calculate the Wasserstein distance between the persistent homology graphs of each time period, and identify the drift of data resource distribution, including: Calculate the Wasserstein distance between the persistent homology graphs in adjacent time periods and generate the distance value between the persistent homology graphs; Based on the distance values ​​between the persistent homology graphs, the distance value change sequence is calculated, and the drift significance score is calculated according to the distance value change sequence according to the preset drift significance calculation logic.

8. The data resource integration system based on cloud computing according to claim 7, characterized in that: The calculation logic of the distance value between persistent homology graphs is: in, t i Time and t j Wasserstein distance between time-persistent homology graphs, is a matching solution set, indicating that The point in All possible combinations of point matches in , ∥xy∥ is the Euclidean distance between two points, ε is a nonlinear amplification parameter used to adjust the sensitivity to distance, To match the cost function, the calculation logic is: Among them, h(x) is the topological level function, which is used to quantify the scale position of the data point x in the persistent homology graph, κ is the weight coefficient, and θ is the time correlation adjustment parameter, which is used to adjust the topological level difference of the matching points; The calculation logic of the distance value change sequence is: Among them, Δ t is a distance value change sequence, which contains the distance values ​​between the persistent homology graphs of all adjacent time periods; The default drift significance calculation logic is: Where S is the drift significance score, T is the number of time periods, which means the total number of time periods compared, that is, the total number of persistent homology graph comparison calculations, Φ(·) is the cumulative distribution function, which is used to measure the significance level of Wasserstein distance, and μ w is the expected value, used as the benchmark value for distance significance test, σ w is a scale parameter used to measure the volatility of distance.

9. The data resource integration system based on cloud computing according to claim 7, characterized in that: According to the drift identification results of data resource distribution, the data resources are classified into: When the drift significance score is greater than the preset drift significance threshold, it means that the data resource drifts significantly. The latest data resource is retained by dichotomy according to the time period, and the drift significance score is recalculated until the drift significance score is less than or equal to the preset drift significance threshold. When the drift significance score is less than or equal to the preset drift significance threshold, it indicates that the data resource drift is not significant, and the data resource is set as a valid resource.

10. The data resource integration system based on cloud computing according to claim 1, characterized in that: It also includes a dimensionality reduction quality evaluation unit, which is used to evaluate the time series input after dimensionality reduction. When the quality metric is greater than a preset quality threshold, it means that the time series input after dimensionality reduction is distorted and cannot represent data resources, and the dimensionality is re-reduced to a quality metric less than or equal to the preset quality threshold. The calculation logic of dimensionality reduction quality evaluation is: Q = ∑ i,j (p ij -q ij ) 2 , Q is the quality measure, p ij is the similarity probability between data point i and data point j in the time series input. The calculation logic is: ∥X scaled,i -X scaled,j ∥ is the Euclidean distance between data point i and data point j in the time series input, σ i is the scale parameter used to calculate the similarity probability, adjusting the width of the similarity distribution around point i, q ij is the similarity probability of data point i and data point j in the time series input after dimensionality reduction. The calculation logic is: Among them, ∥Y i -Y j ∥ is the Euclidean distance between data point i and data point j in the time series input after dimensionality reduction.

Images