Intelligent big data acquisition and analysis method and system based on deep learning
By using deep learning technology, combined with immune algorithms, topological data analysis, graph neural networks, and counterfactual reasoning, the problems of complexity, accuracy, and real-time performance in data acquisition and analysis have been solved, achieving efficient and accurate data processing and intelligent decision support.
Patent Information
- Application Number
- CN202411075277.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing technologies suffer from problems in data acquisition and analysis, such as data processing complexity, insufficient analytical accuracy, difficulty in meeting real-time requirements, and high difficulty for users to understand and apply them.
This paper adopts a deep learning-based intelligent big data collection and analysis method, uses an immune algorithm for anomaly detection and correction, utilizes multi-scale convolutional networks for data denoising and standardization, combines topological data analysis and persistent cohomology methods to extract high-dimensional topological structures, employs graph neural networks and graph embedding methods for high-order relation modeling, utilizes non-negative matrix factorization and tensor factorization for multi-perspective data integration, and achieves adaptive model adjustment through counterfactual reasoning and reinforcement learning, providing a high-dimensional data visualization tool.
It improves the accuracy and efficiency of data preprocessing, significantly enhances the accuracy and depth of data analysis, meets the needs of real-time data processing and analysis, and reduces the difficulty for users to understand and apply.
Smart Images

Figure CN119004072A_ABST
Abstract
Description
[0001] The present invention relates to the field of deep learning and big data technology, and in particular to an intelligent big data collection and analysis method and system based on deep learning. Background Art
[0002] With the rapid development of big data and Internet of Things technologies, data collection and analysis have become an important means for various industries to improve efficiency and optimize decision-making. In existing technologies, data collection usually relies on configuring sensors, databases, and API interfaces to collect multi-source data through these channels. However, this method often seems to be unable to cope with problems such as data diversity, large data volume, and data quality.
[0003] First, data processing complexity is one of the main problems in the existing technology. Existing data acquisition systems usually collect data from different data sources by configuring sensors, databases, and API interfaces. These data sources include IoT devices, sensor networks, and database systems. However, the data formats and standards of different data sources are not unified, resulting in high complexity in data integration and processing. Data quality and consistency are difficult to guarantee, especially in the data preprocessing stage, there are problems such as inconsistent data formats, data redundancy, and missing data. These problems make the data preprocessing process complex and time-consuming.
[0004] Secondly, insufficient analysis accuracy is another significant flaw. Traditional data analysis methods mainly rely on manual feature engineering and simple statistical analysis methods. These methods are difficult to fully utilize the high-dimensional features and complex relationships in the data, and the analysis models often fail to capture the deep patterns and features in the data, resulting in inaccurate and in-depth analysis results. Especially in the face of noise and outliers, the existing technology lacks efficient anomaly detection and data correction methods, and data quality is difficult to guarantee.
[0005] Third, lack of real-time performance is also a major challenge faced by existing technologies. The widespread use of IoT devices and sensors has led to an explosive growth in the amount of data, and traditional data processing methods are unable to efficiently process such a large amount of data within a reasonable time. In many application scenarios, real-time processing and analysis of data are crucial, but existing technologies have a certain delay in processing real-time data and cannot meet the real-time requirements of key application scenarios.
[0006] Fourth, it is difficult for users to understand and apply. Complex data analysis results are often difficult to understand and apply intuitively, and existing technologies lack effective visualization tools to help users intuitively understand and use analysis results. This makes users face high technical barriers and difficulty in understanding when making data-driven decisions.
[0007] In summary, the existing technologies have many defects in data collection and analysis, including data processing complexity, insufficient analysis accuracy, difficulty in meeting real-time requirements, and difficulty in user understanding and application. The present invention proposes an intelligent big data collection and analysis method and system based on deep learning, which integrates multiple advanced technical means, such as immune algorithm, topological data analysis, graph neural network, non-negative matrix decomposition and counterfactual reasoning, to provide efficient, accurate and intelligent data collection and analysis solutions to meet the diverse needs in modern big data applications.
[0008] Therefore, how to provide an intelligent big data collection and analysis method and system based on deep learning is a problem that technical personnel in this field urgently need to solve. Summary of the invention
[0009] One purpose of the present invention is to propose an intelligent big data collection and analysis method and system based on deep learning, which effectively solves the problems of data processing complexity, insufficient analysis accuracy, difficulty in meeting real-time requirements, and difficulty for users to understand and apply.
[0010] According to an embodiment of the present invention, a method and system for intelligent big data collection and analysis based on deep learning includes the following steps:
[0011] S1. Obtain data by configuring sensors, databases, API interfaces and IoT devices;
[0012] S2, use immune algorithm for anomaly detection and correction, and use multi-scale convolutional network for data denoising and standardization;
[0013] S3, using topological data analysis and persistent homology methods to extract persistent features from the data and identify high-dimensional topological structures in the data;
[0014] S4, using diffusion mapping and manifold learning techniques to perform nonlinear dimensionality reduction and analysis on the extracted high-dimensional features to capture the complex structures and patterns in the data;
[0015] S5. Use graph neural networks and graph embedding methods to build complex relationship graphs in data and model and analyze high-order relationships;
[0016] S6. Use non-negative matrix decomposition and tensor decomposition methods to collaboratively train data from different perspectives and integrate multi-perspective information;
[0017] S7. Use counterfactual reasoning methods to achieve adaptive adjustment of the model in new data environments, and combine reinforcement learning for intelligent decision support;
[0018] S8. Utilize heterogeneous information network embedding methods to provide visualization tools for high-dimensional data and generate analysis reports and intelligent decision-making recommendations.
[0019] Optionally, the S1 includes the following steps:
[0020] S11. Configure sensors, databases, API interfaces and IoT devices to build a multi-source data collection system;
[0021] S12. Extract data from different databases through API interfaces and integrate information from various databases;
[0022] S13. Use IoT devices to collect sensor data in real time and collect information such as environment and device status;
[0023] S14. Integrate multi-source data and build a comprehensive dataset.
[0024] Optionally, S2 includes the following steps:
[0025] S21. Use immune algorithm for anomaly detection and correction:
[0026] Define the antigen data set A = {a1, a2, ..., a n} and antibody library B = {b1, b2, …, b m}, where a i represents the i-th data point, b j represents the jth antibody;
[0027] Calculate the affinity matrix AFF:
[0028]
[0029] Among them, ∥a i -b j ∥ represents data point a i and antibody b j The Euclidean distance between them, σ is the scale parameter of the affinity function;
[0030] Screen high-affinity antibodies based on the affinity matrix and generate candidate correction data points through the clone selection algorithm:
[0031]
[0032] Among them, A′ i is the corrected data point;
[0033] S22, use multi-scale convolutional network to denoise data, using multi-scale convolution kernel K s Perform multi-level feature extraction on data X:
[0034] Fs =K s *X+b s ;
[0035] Among them, * represents the convolution operation, K s represents the convolution kernel of scale s, b s is the bias term, F s is the feature map at scale s;
[0036] Through the adaptive feature fusion strategy, the multi-scale feature map F s Perform weighted fusion to generate denoised data X′:
[0037]
[0038] Among them, α s Represents the feature weight coefficient of scale s calculated by the attention mechanism:
[0039]
[0040] Among them, β s is the feature importance score of scale s;
[0041] S23, standardize the denoised data and calculate the mean μ and standard deviation μ of the denoised data:
[0042]
[0043] Where n is the number of data points, X′ i represents the i-th data point after denoising;
[0044] S24. Standardize the data:
[0045]
[0046] Among them, X″ i is the i-th data point after standardization.
[0047] Optionally, S3 includes the following steps:
[0048] S31. Construct the topological space of data and transform the data X″ i Represented as a point cloud P = {x1, x2, ..., x n};
[0049] S32. Use random geometric complexes to perform topological construction on the point cloud P and generate random geometric complexes R(r):
[0050]
[0051] Among them, r is the connection radius, ||x i -x j || represents the data point x i and x j The Euclidean distance between
[0052] S33. Calculate the persistent homology group under different connection radius r, and obtain the homology group H under each parameter k (R(r)):
[0053]
[0054] Where k represents the dimension of homology, is the boundary operator;
[0055] S34. Use topology optimization entropy to optimize the persistent strips and record the birth and death relationship of topological features under different parameters ∈:
[0056] Barcode = {(b i ,d i )∣0≤b i <d i ≤∞};
[0057] Among them, b i is the birth parameter of topological features, d i is the extinction parameter of the topological feature; the topology optimization entropy is defined as follows:
[0058]
[0059] Among them, σ is the probability density of the i-th point, and λ is the regularization parameter;
[0060] S35. Convert the persistent strip into a persistent graph to record the persistence of topological features:
[0061] Diagram={(b i ,d i )∣0≤b i <d i ≤∞};
[0062] Among them, each point (b i ,d i ) represents the persistence of a topological feature;
[0063] S36. Use high-dimensional homology convolutional neural networks to learn persistence graphs and extract deep representations of topological features:
[0064] z = HD-HCNN (Diagram);
[0065] Where z is the extracted deep topological feature vector, HD-HCNN represents HD-HCNN, and the high-dimensional homology convolutional neural network uses the following convolution operation:
[0066]
[0067] Among them, h (l) represents the features of the lth layer, represents the weight matrix of the lth layer at the sth scale, A s is the adjacency matrix under scale s, b (l) is the bias, σ is the nonlinear activation function;
[0068] S37. Generate new topological structures using generative adversarial networks:
[0069] G′=GAN(z);
[0070] Among them, G′ is the generated graph structure, and GAN represents the generative adversarial network;
[0071] S38. Perform feature optimization and dimensionality reduction on the generated graph structure, and use high-dimensional space nonlinear projection technology to map high-dimensional topological features to low-dimensional space:
[0072] e=tanh(W p G′+b p );
[0073] Among them, W p is the projection weight matrix, b p is the projection bias, tanh is the activation function;
[0074] S39. Use low-dimensional topological feature vector e for data classification and clustering:
[0075] Class = softmax(W c eb c );
[0076] Among them, W c is the classification weight matrix, b c is the classification bias vector.
[0077] Optionally, S4 includes the following steps:
[0078] S41. Construct similarity matrix W and define data point x i and x j The similarity between ij :
[0079]
[0080] Among them, ∥x i -xj ∥ represents data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter;
[0081] S42, construct an angle matrix D, whose element d ii is the row sum of the similarity matrix W:
[0082]
[0083] S43. Calculate the normalized Laplace matrix L:
[0084] L=ID -1 / 2 WD -1 / 2 ;
[0085] Where I is the identity matrix;
[0086] S44, perform eigenvalue decomposition to solve the eigenvalues and eigenvectors of the normalized Laplace matrix L:
[0087] Lφ k =λ k φ k ;
[0088] Among them, λ k and φ k are the kth eigenvalue and eigenvector respectively;
[0089] S45. Use the eigenvectors corresponding to the first m smallest non-zero eigenvalues to construct a dimensionality reduction mapping:
[0090] y i =(φ1(x i ),φ2(x i ),…,φ m (x i ));
[0091] Among them, y i is the data point after dimensionality reduction;
[0092] S46. Introducing high-dimensional nonlinear metrics D NL :
[0093]
[0094] Among them, F t (x i ) represents the data point x at time step t i Nonlinear transformation of :
[0095] F t (x i )=ReLU(W t yi +b t );
[0096] Among them, W t and b t are the weight matrix and bias at time step t, respectively, and ReLU is the activation function;
[0097] S47. Use variational graph neural network to learn and optimize the features after dimensionality reduction:
[0098] q(z|y)=N(z;μ(y),σ 2 (y));
[0099] Among them, μ and σ are the mean and variance functions in VGNN, defined as:
[0100] μ(y)=W μ y+b μ ;
[0101] σ(y)=exp(W σ y+b σ );
[0102] Among them, W μ , W σ is the weight matrix, b μ 、b σ is bias;
[0103] S48. Optimize the feature representation after dimensionality reduction through variational inference:
[0104] z i =μ(y i )+σ(y i )⊙∈;
[0105] Among them, ∈ is a standard normal distribution random variable, ⊙ represents the element-by-element product;
[0106] S49. Use the multi-level high-order spectral clustering method to cluster the final features, and define the optimization objective function as:
[0107]
[0108] Among them, L is the number of clustering layers, K is the number of clusters in each layer, is the kth cluster in the lth layer, For data point x i Belongs to cluster The probability of , β is the regularization parameter, n (l) is the total number of data points in layer l.
[0109] Optionally, S5 includes the following steps:
[0110] S51. Build a data relationship graph G, which contains a node set V and an edge set E. Define node v i and v j The edge weight w between ij for:
[0111]
[0112] Among them, ∥x i -x j ∥ represents data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter;
[0113] S52, construct an adjacency matrix A, whose element a ij Represents node v i and v j The connection relationship between:
[0114]
[0115] S53, use Graph Convolutional Network (GCN) to extract features from graph structure data and define node features H (0) For input data X, the node features are updated by the following iterations:
[0116]
[0117] in, is the adjacency matrix with self-loops added, for The degree matrix, W (l) is the weight matrix of the lth layer, σ is the activation function, H (l) is the node feature of the lth layer;
[0118] S54. Introduce a self-supervised learning mechanism to further optimize graph embedding through a contrast loss function. Define the contrast loss as:
[0119]
[0120] Among them, h i and are the feature vectors of the positive sample pairs, is the feature vector of the negative sample pair, sim(·) is the similarity measurement function;
[0121] S55. Use graph embedding technology to map high-dimensional graph features to low-dimensional space:
[0122] Z = GraphEmbed(H (L) );
[0123] Among them, H (L) is the last layer output of GCN, Z is the low-dimensional embedding representation, and GraphEmbed is the graph embedding function;
[0124] S56. Use graph variational autoencoder to further optimize graph embedding representation and define variational loss function as:
[0125]
[0126] Where q(z|x) is the approximate posterior distribution, p(z) is the prior distribution, and KL(·||·) is the KL divergence;
[0127] S57. Introduce a multi-view graph convolutional network to capture the node relationship under different viewpoints. The specific definition is as follows:
[0128]
[0129] in, is the adjacency matrix of the vth perspective, for The degree matrix, α v is the weight coefficient of the vth viewing angle, is the weight matrix of the vth view in the lth layer;
[0130] S58. Finally, the output features of the multi-view graph convolutional network are used to perform relationship modeling and analysis to obtain a high-dimensional representation of the nodes:
[0131] Z (MV) =H (L) ;
[0132] Among them, Z (MV) It is the final output feature of the multi-view graph convolutional network;
[0133] S59. Cluster the high-dimensional representation of the nodes through high-order spectral clustering, and define the optimization objective function as:
[0134]
[0135] Among them, K is the number of clusters, C k is the kth cluster, p(v i ∣C k ) is the node v i Belongs to cluster C k The probability of , γ is the regularization parameter, and n is the total number of nodes.
[0136] Optionally, the S6 includes the following steps:
[0137] S61, constructing a data matrix X and a multi-view tensor T, where X represents a basic feature matrix of the data, and T represents a multi-view tensor;
[0138] S62. Perform non-negative matrix decomposition to decompose the data matrix X into two non-negative matrices W and H, which are defined as follows:
[0139] X≈WH;
[0140] Among them, W ≥ 0 is the basis matrix, H ≥ 0 is the coefficient matrix; optimization is performed by minimizing the following loss function:
[0141]
[0142] Among them, ||·|| F is the Frobenius norm, λ is the regularization parameter;
[0143] S63. Construct a multi-view tensor T, whose elements T ijk Indicates the relationship between the jth feature and the kth feature at the i-th perspective;
[0144] S64, perform high-order tensor decomposition to decompose the multi-view tensor T into a set of factor matrices {A i}, defined as follows:
[0145]
[0146] Among them, λ r is the weight coefficient, A ir is the rth factor matrix, represents the tensor product; it is optimized by minimizing the following loss function:
[0147]
[0148] S65. Introduce a collaborative training mechanism, integrate multi-view information by jointly optimizing non-negative matrix decomposition and high-order tensor decomposition, and define the joint loss function as:
[0149] L Joint =L NMF +αL HOTD ;
[0150] Among them, α is the weight parameter of collaborative training;
[0151] S66. Use variational Bayesian inference to optimize the joint model and define the variational posterior distribution q(θ) as:
[0152]
[0153] Among them, θ represents the model parameters, φ iis the variational parameter; the optimization is performed by minimizing the following evidence lower bound:
[0154] L VBI =E q(θ) [logp(X,T∣θ)]-KL(q(θ)∥p(θ));
[0155] Among them, KL(·∥·) is the KL divergence, p(θ) is the prior distribution;
[0156] S67. Adaptive sparse regularization is introduced in variational Bayesian inference, and the adaptive sparse regularization term is defined as:
[0157]
[0158] Among them, η ij is the adaptive sparse parameter, ∈ is a small value to prevent division by zero;
[0159] S68, perform multi-view information fusion and use the fused feature matrix \(Z\) for data analysis:
[0160]
[0161] Among them, Z is the fused feature representation;
[0162] S69. Perform cluster analysis on the fused features through high-order spectral clustering, and define the clustering objective function as:
[0163]
[0164] Among them, K is the number of clusters, C k is the kth cluster, p(z i ∣C k ) is the feature z i Belongs to cluster C k The probability of , β is the regularization parameter, n is the total number of features, a ij is the similarity matrix element, and γ is the similarity regularization parameter.
[0165] Optionally, the S7 includes the following steps:
[0166] S71. Construct a counterfactual reasoning model and define the counterfactual sample generation process. Given input data x and target output y, generate counterfactual samples x′ by perturbing the original input:
[0167] x′=x+δ;
[0168] Among them, δ is the perturbation amount, which is optimized by minimizing the following loss function:
[0169] L CF =∥f(x)-y∥2 +λ∥δ∥ 2 ;
[0170] Where f(x) is the predicted output of the model and λ is the regularization parameter;
[0171] S72. Introduce an adaptive adjustment mechanism in the counterfactual reasoning model to achieve adaptive adjustment of the model in the new data environment by minimizing the difference between the counterfactual samples and the original samples:
[0172] L Adj =L CF +α∥x′-x∥ 2 ;
[0173] Among them, α is the adaptive adjustment weight parameter;
[0174] S73. Combine the reinforcement learning (RL) method to build a policy gradient-based reinforcement learning model; define the state s, action a and reward r as:
[0175] s t = x′ t ,a t =π(s t ),r t =R(s t ,a t );
[0176] Among them, π(s t ) is the policy function, R(s t ,a t ) is the reward function;
[0177] S74. Optimize the policy function π using the policy gradient method by minimizing the following loss function:
[0178]
[0179] Among them, γ is the discount factor;
[0180] S75. Introduce counterfactual reasoning results into the reinforcement learning model, optimize the strategy function through the adaptively adjusted counterfactual samples, and define the optimization goal as:
[0181] L CRL =L PG +βL Adj ;
[0182] Among them, β is the counterfactual reasoning weight parameter;
[0183] S76. Combine the multi-task learning (MTL) method to build a multi-task counterfactual reinforcement learning model; define the multi-task loss function as:
[0184]
[0185] Where M is the number of tasks, is the counterfactual reinforcement learning loss for the i-th task;
[0186] S77. Define attention weight α i for:
[0187]
[0188] Among them, e i is the attention score of the i-th task, which is optimized by minimizing the following weighted loss function:
[0189]
[0190] S78. Implement an intelligent decision support system, make real-time decisions through a multi-task counterfactual reinforcement learning model, and define the final decision function D as:
[0191] D = argmax a π(s,a);
[0192] Where s is the current state, a is the optional action, and π(s,a) is the action probability output by the policy function;
[0193] S79. Introduce high-order counterfactual reasoning to generate counterfactual samples at multiple levels:
[0194] x (k+1) =x (k) +δ (k)
[0195] Among them, δ (k) is the perturbation amount of the kth layer, and the multi-level loss function is:
[0196]
[0197] S710, introduce complex reinforcement learning mechanism combined with Gaussian process to define the strategy function:
[0198] π(s t )=GP(s t ,θ);
[0199] Among them, GP is the Gaussian process model, θ is the model parameter, and optimization is performed by minimizing the following loss function:
[0200]
[0201] S711. Introducing complex regularization terms in the optimization process:
[0202]
[0203] in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature;
[0204] S712, adaptive moment estimation using adaptive optimization algorithm:
[0205]
[0206] Among them, η is the learning rate, m t is the first-order moment estimate of the gradient, v t is the second moment estimate of the gradient.
[0207] Optionally, the S8 includes the following steps:
[0208] S81. Construct a heterogeneous information network and define a graph structure G = (V, E) containing different types of nodes and edges, where the node set V and the edge set E are expressed as:
[0209] V={v1,v2,…,v n};
[0210] E={e1,e2,…,e m};
[0211] Among them, v i are of different types T v , edge i are different relationship types T e ;
[0212] S82. Construct a meta-path to represent different types of relationships in a heterogeneous information network, and define the meta-path P as:
[0213] P=(A1,R1,A2,R2,…,R k-1 ,A k );
[0214] Among them, A i Indicates the node type, R i Indicates the relationship type;
[0215] S83. Perform a random walk on each meta-path P to generate sequence data S:
[0216] S=(v1,v2,…,v l);
[0217] Among them, v i is a node, and the sequence length is l;
[0218] S84. Map the nodes in the heterogeneous information network to a low-dimensional vector space through a multimodal embedding method; define the embedding matrix M as:
[0219] M = MM-HIN2Vec(G);
[0220] Among them, MM-HIN2Vec is an embedding method for multimodal heterogeneous information networks, and G is a graph structure;
[0221] S85. Use graph convolutional variational autoencoder to extract features from the embedded low-dimensional vector and define the node feature matrix H as:
[0222] H = GCVAE(M);
[0223] Among them, GCVAE is the graph convolutional variational autoencoder model, and M is the embedding matrix;
[0224] S86. Use graph generation adversarial network to generate and train features, and define the loss functions of the generator and discriminator as:
[0225]
[0226] Among them, G is the generator, D is the discriminator, and p z (z) is the noise distribution, p data (x) is the data distribution;
[0227] S87, introduce complex regularization terms:
[0228]
[0229] in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature;
[0230] S88, visualization technology, visualizes the embedded low-dimensional vector and classification clustering results:
[0231] V = DL-Vis(H);
[0232] Among them, DL-Vis is a visualization algorithm based on deep learning, which is used to map high-dimensional data into two-dimensional space for visualization;
[0233] S89. Generate analysis reports. Combine visualization results and model outputs to generate reports containing data insights and decision recommendations:
[0234]
[0235] in, is the predicted label set, and C is the clustering result set.
[0236] The beneficial effects of the present invention are:
[0237] The present invention adopts immune algorithm to detect and correct anomalies, and performs data denoising and standardization through multi-scale convolutional network, which improves the accuracy and efficiency of data preprocessing and solves the problems of poor data noise and outlier processing.
[0238] The present invention utilizes topological data analysis and persistent homology methods to extract persistent features from data and identify high-dimensional topological structures. It combines diffusion mapping and manifold learning techniques to perform nonlinear dimensionality reduction and analysis on high-dimensional features, significantly improving the accuracy and depth of data analysis.
[0239] The present invention adopts graph neural network and graph embedding methods to construct complex relationship graphs, perform high-order relationship modeling and analysis, and combines non-negative matrix decomposition and tensor decomposition methods to achieve collaborative training and integration of multi-perspective data, comprehensively improving the depth and breadth of data analysis.
[0240] The present invention uses the counterfactual reasoning method to achieve adaptive adjustment of the model in a new data environment, and combines reinforcement learning for intelligent decision support, thus meeting the needs of real-time data processing and analysis.
[0241] The present invention ensures the quality and consistency of data by configuring sensors, databases, API interfaces and Internet of Things devices, and significantly improves the accuracy and efficiency of data collection. BRIEF DESCRIPTION OF THE DRAWINGS
[0242] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:
[0243] Figure 1 A flowchart of an intelligent big data collection and analysis method and system based on deep learning proposed by the present invention;
[0244] Figure 2 A schematic diagram of topological data analysis and high-dimensional feature extraction of a deep learning-based intelligent big data collection and analysis method and system proposed in the present invention;
[0245] Figure 3 Schematic diagram of graph neural network and multi-perspective analysis of a deep learning-based intelligent big data collection and analysis method and system proposed in the present invention. DETAILED DESCRIPTION
[0246] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.
[0247] refer to Figure 1-3 , an intelligent big data collection and analysis method and system based on deep learning, comprising the following steps:
[0248] S1. Obtain data by configuring sensors, databases, API interfaces and IoT devices;
[0249] S2, use immune algorithm for anomaly detection and correction, and use multi-scale convolutional network for data denoising and standardization;
[0250] S3, using topological data analysis and persistent homology methods to extract persistent features from the data and identify high-dimensional topological structures in the data;
[0251] S4, using diffusion mapping and manifold learning techniques to perform nonlinear dimensionality reduction and analysis on the extracted high-dimensional features to capture the complex structures and patterns in the data;
[0252] S5. Use graph neural networks and graph embedding methods to build complex relationship graphs in data and model and analyze high-order relationships;
[0253] S6. Use non-negative matrix decomposition and tensor decomposition methods to collaboratively train data from different perspectives and integrate multi-perspective information;
[0254] S7. Use counterfactual reasoning methods to achieve adaptive adjustment of the model in new data environments, and combine reinforcement learning for intelligent decision support;
[0255] S8. Utilize heterogeneous information network embedding methods to provide visualization tools for high-dimensional data and generate analysis reports and intelligent decision-making recommendations.
[0256] In this implementation, S1 includes the following steps:
[0257] S11. Configure sensors, databases, API interfaces and IoT devices to build a multi-source data collection system;
[0258] S12. Extract data from different databases through API interfaces and integrate information from various databases;
[0259] S13. Use IoT devices to collect sensor data in real time and collect information such as environment and device status;
[0260] S14. Integrate multi-source data and build a comprehensive dataset.
[0261] In this implementation, S2 includes the following steps:
[0262] S21. Use immune algorithm for anomaly detection and correction:
[0263] Define the antigen data set A = {a1, a2, ..., a n} and antibody library B = {b1, b2, …, b m}, where a i represents the i-th data point, b j represents the jth antibody;
[0264] Calculate the affinity matrix AFF:
[0265]
[0266] Among them, ∥a i -b j ∥ represents data point a i and antibody b j The Euclidean distance between them, σ is the scale parameter of the affinity function;
[0267] Screen high-affinity antibodies based on the affinity matrix and generate candidate correction data points through the clone selection algorithm:
[0268]
[0269] Among them, A′ i is the corrected data point;
[0270] S22, use multi-scale convolutional network to denoise data, using multi-scale convolution kernel K s Perform multi-level feature extraction on data X:
[0271] F s =K s *X+b s ;
[0272] Among them, * represents the convolution operation, K s represents the convolution kernel of scale s, b s is the bias term, F s is the feature map at scale s;
[0273] Through the adaptive feature fusion strategy, the multi-scale feature map F s Perform weighted fusion to generate denoised data X′:
[0274]
[0275] Among them, α s Represents the feature weight coefficient of scale s calculated by the attention mechanism:
[0276]
[0277] Among them, β s is the feature importance score of scale s;
[0278] S23, standardize the denoised data and calculate the mean μ and standard deviation μ of the denoised data:
[0279]
[0280] Where n is the number of data points, X′ i represents the i-th data point after denoising;
[0281] S24. Standardize the data:
[0282]
[0283] Among them, X″ i is the i-th data point after standardization.
[0284] In this implementation, S3 includes the following steps:
[0285] S31. Construct the topological space of data and transform the data X″ i Represented as a point cloud P = {x1, x2, ..., x n};
[0286] S32. Use random geometric complexes to perform topological construction on the point cloud P and generate random geometric complexes R(r):
[0287]
[0288] Among them, r is the connection radius, ||x i -x j || represents the data point x i and x j The Euclidean distance between
[0289] S33. Calculate the persistent homology group under different connection radius r, and obtain the homology group H under each parameter k (R(r)):
[0290]
[0291] Where k represents the dimension of homology, is the boundary operator;
[0292] S34. Use topology optimization entropy to optimize the persistent strips and record the birth and death relationship of topological features under different parameters ∈:
[0293] Barcode = {(b i ,d i )∣0≤b i <d i ≤∞};
[0294] Among them, b i is the birth parameter of topological features, d i is the extinction parameter of the topological feature; the topology optimization entropy is defined as follows:
[0295]
[0296] Among them, σ is the probability density of the i-th point, and λ is the regularization parameter;
[0297] S35. Convert the persistent strip into a persistent graph to record the persistence of topological features:
[0298] Diagram={(b i ,d i )∣0≤b i <d i ≤∞};
[0299] Among them, each point (b i ,d i ) represents the persistence of a topological feature;
[0300] S36. Use high-dimensional homology convolutional neural networks to learn persistence graphs and extract deep representations of topological features:
[0301] z = HD-HCNN (Diagram);
[0302] Among them, z is the extracted deep topological feature vector, HD-HCNN represents the high-dimensional homology convolutional neural network, and the high-dimensional homology convolutional neural network uses the following convolution operation:
[0303]
[0304] Among them, h (l) represents the features of the lth layer, represents the weight matrix of the lth layer at the sth scale, A s is the adjacency matrix under scale s, b (l) is the bias, σ is the nonlinear activation function;
[0305] S37. Generate new topological structures using generative adversarial networks:
[0306] G′=GAN(z);
[0307] Among them, G′ is the generated graph structure, and GAN represents the generative adversarial network;
[0308] S38. Perform feature optimization and dimensionality reduction on the generated graph structure, and use high-dimensional space nonlinear projection technology to map high-dimensional topological features to low-dimensional space:
[0309] e=tanh(W p G′+b p );
[0310] Among them, W p is the projection weight matrix, b p is the projection bias, tanh is the activation function;
[0311] S39. Use low-dimensional topological feature vector e for data classification and clustering:
[0312] Class = softmax(W c eb c );
[0313] Among them, W c is the classification weight matrix, b c is the classification bias vector.
[0314] In this implementation, S4 includes the following steps:
[0315] S41. Construct similarity matrix W and define data point x i and x j The similarity between ij :
[0316]
[0317] Among them, ∥x i -x j ∥ represents data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter;
[0318] S42, construct an angle matrix D, whose element d ii is the row sum of the similarity matrix W:
[0319]
[0320] S43. Calculate the normalized Laplace matrix L:
[0321] L=ID -1 / 2 WD -1 / 2 ;
[0322] Where I is the identity matrix;
[0323] S44, perform eigenvalue decomposition to solve the eigenvalues and eigenvectors of the normalized Laplace matrix L:
[0324] Lφ k =λ k φ k ;
[0325] Among them, λ k and φ k are the kth eigenvalue and eigenvector respectively;
[0326] S45. Use the eigenvectors corresponding to the first m smallest non-zero eigenvalues to construct a dimensionality reduction mapping:
[0327] y i =(φ1(x i ),φ2(x i ),…,φ m (x i ));
[0328] Among them, y i is the data point after dimensionality reduction;
[0329] S46. Introducing high-dimensional nonlinear metrics D NL :
[0330]
[0331] Among them, F t (x i ) represents the data point x at time step t i Nonlinear transformation of :
[0332] F t (x i )=ReLU(W t y i +b t );
[0333] Among them, W t and b t are the weight matrix and bias at time step t, respectively, and ReLU is the activation function;
[0334] S47. Use variational graph neural network to learn and optimize the features after dimensionality reduction:
[0335] q(z|y)=N(z;μ(y),σ 2 (y));
[0336] Among them, μ and σ are the mean and variance functions in VGNN, defined as:
[0337] μ(y)=W μ y+bμ ;
[0338] σ(y)=exp(W σ y+b σ );
[0339] Among them, W μ , W σ is the weight matrix, b μ 、b σ is bias;
[0340] S48. Optimize the feature representation after dimensionality reduction through variational inference:
[0341] z i =μ(y i )+σ(y i )⊙∈;
[0342] Among them, ∈ is a standard normal distribution random variable, ⊙ represents the element-by-element product;
[0343] S49. Use the multi-level high-order spectral clustering method to cluster the final features, and define the optimization objective function as:
[0344]
[0345] Among them, L is the number of clustering layers, K is the number of clusters in each layer, is the kth cluster in the lth layer, For data point x i Belongs to cluster The probability of , β is the regularization parameter, n (l) is the total number of data points in layer l.
[0346] In this implementation, S5 includes the following steps:
[0347] S51. Build a data relationship graph G, which contains a node set V and an edge set E. Define node v i and v j The edge weight w between ij for:
[0348]
[0349] Among them, ||x i -x j || represents the data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter;
[0350] S52, construct an adjacency matrix A, whose element a ij Represents node v i and v jThe connection relationship between:
[0351]
[0352] S53, use Graph Convolutional Network (GCN) to extract features from graph structure data and define node features H (0) For input data X, the node features are updated by the following iterations:
[0353]
[0354] in, is the adjacency matrix with self-loops added, for The degree matrix, W (l) is the weight matrix of the lth layer, σ is the activation function, H (l) is the node feature of the lth layer;
[0355] S54. Introduce a self-supervised learning mechanism to further optimize graph embedding through a contrast loss function. Define the contrast loss as:
[0356]
[0357] Among them, h i and are the feature vectors of the positive sample pairs, is the feature vector of the negative sample pair, sim(·) is the similarity measurement function;
[0358] S55. Use graph embedding technology to map high-dimensional graph features to low-dimensional space:
[0359] Z = GraphEmbed(H (L) );
[0360] Among them, H (L) is the last layer output of GCN, Z is the low-dimensional embedding representation, and GraphEmbed is the graph embedding function;
[0361] S56. Use graph variational autoencoder to further optimize graph embedding representation and define variational loss function as:
[0362]
[0363] Where q(z|x) is the approximate posterior distribution, p(z) is the prior distribution, and KL(·||·) is the KL divergence;
[0364] S57. Introduce a multi-view graph convolutional network to capture the node relationship under different viewpoints. The specific definition is as follows:
[0365]
[0366] in, is the adjacency matrix of the vth perspective, for The degree matrix, α v is the weight coefficient of the vth viewing angle, is the weight matrix of the vth view in the lth layer;
[0367] S58. Finally, the output features of the multi-view graph convolutional network are used to perform relationship modeling and analysis to obtain a high-dimensional representation of the nodes:
[0368] Z (MV) =H (L) ;
[0369] Among them, Z (MV) It is the final output feature of the multi-view graph convolutional network;
[0370] S59. Cluster the high-dimensional representation of the nodes through high-order spectral clustering, and define the optimization objective function as:
[0371]
[0372] Among them, K is the number of clusters, C k is the kth cluster, p(v i ∣C k ) is the node v i Belongs to cluster C k The probability of , γ is the regularization parameter, and n is the total number of nodes.
[0373] In this implementation, S6 includes the following steps:
[0374] S61, constructing a data matrix X and a multi-view tensor T, where X represents a basic feature matrix of the data, and T represents a multi-view tensor;
[0375] S62. Perform non-negative matrix decomposition to decompose the data matrix X into two non-negative matrices W and H, which are defined as follows:
[0376] X≈WH;
[0377] Among them, W ≥ 0 is the basis matrix, H ≥ 0 is the coefficient matrix; optimization is performed by minimizing the following loss function:
[0378]
[0379] Among them, ∥·∥ F is the Frobenius norm, λ is the regularization parameter;
[0380] S63. Construct a multi-view tensor T, whose elements T ijkIndicates the relationship between the jth feature and the kth feature at the i-th perspective;
[0381] S64, perform high-order tensor decomposition to decompose the multi-view tensor T into a set of factor matrices {A i}, defined as follows:
[0382]
[0383] Among them, λ r is the weight coefficient, A ir is the rth factor matrix, represents the tensor product; it is optimized by minimizing the following loss function:
[0384]
[0385] S65. Introduce a collaborative training mechanism, integrate multi-view information by jointly optimizing non-negative matrix decomposition and high-order tensor decomposition, and define the joint loss function as:
[0386] L Joint =L NMF +αL HOTD ;
[0387] Among them, α is the weight parameter of collaborative training;
[0388] S66. Use variational Bayesian inference to optimize the joint model and define the variational posterior distribution q(θ) as:
[0389]
[0390] Among them, θ represents the model parameters, φ i is the variational parameter; the optimization is performed by minimizing the following evidence lower bound:
[0391] L VBI =E q(θ) [logp(X,T∣θ)]-KL(q(θ)∥p(θ));
[0392] Among them, KL(·∥·) is the KL divergence, p(θ) is the prior distribution;
[0393] S67. Adaptive sparse regularization is introduced in variational Bayesian inference, and the adaptive sparse regularization term is defined as:
[0394]
[0395] Among them, η ij is the adaptive sparse parameter, ∈ is a small value to prevent division by zero;
[0396] S68, perform multi-view information fusion and use the fused feature matrix \(Z\) for data analysis:
[0397]
[0398] Among them, Z is the fused feature representation;
[0399] S69. Perform cluster analysis on the fused features through high-order spectral clustering, and define the clustering objective function as:
[0400]
[0401] Among them, K is the number of clusters, C k is the kth cluster, p(z i ∣C k ) is the feature z i Belongs to cluster C k The probability of , β is the regularization parameter, n is the total number of features, a ij is the similarity matrix element, and γ is the similarity regularization parameter.
[0402] In this implementation, S7 includes the following steps:
[0403] S71. Construct a counterfactual reasoning model and define the counterfactual sample generation process. Given input data x and target output y, generate counterfactual samples x′ by perturbing the original input:
[0404] x′=x+δ;
[0405] Among them, δ is the perturbation amount, which is optimized by minimizing the following loss function:
[0406] L CF =∥f(x)-y∥ 2 +λ∥δ∥ 2 ;
[0407] Where f(x) is the predicted output of the model and λ is the regularization parameter;
[0408] S72. Introduce an adaptive adjustment mechanism in the counterfactual reasoning model to achieve adaptive adjustment of the model in the new data environment by minimizing the difference between the counterfactual samples and the original samples:
[0409] L Adj =L CF +α∥x′-x∥ 2 ;
[0410] Among them, α is the adaptive adjustment weight parameter;
[0411] S73. Combine the reinforcement learning (RL) method to build a policy gradient-based reinforcement learning model; define the state s, action a and reward r as:
[0412] s t = x′ t ,a t =π(s t ),r t =R(s t ,a t );
[0413] Among them, π(s t ) is the policy function, R(s t ,a t ) is the reward function;
[0414] S74. Optimize the policy function π using the policy gradient method by minimizing the following loss function:
[0415]
[0416] Among them, γ is the discount factor;
[0417] S75. Introduce counterfactual reasoning results into the reinforcement learning model, optimize the strategy function through the adaptively adjusted counterfactual samples, and define the optimization goal as:
[0418] L CRL =L PG +βL Adj ;
[0419] Among them, β is the counterfactual reasoning weight parameter;
[0420] S76. Combine the multi-task learning (MTL) method to build a multi-task counterfactual reinforcement learning model; define the multi-task loss function as:
[0421]
[0422] Where M is the number of tasks, is the counterfactual reinforcement learning loss for the i-th task;
[0423] S77. Define attention weight α i for:
[0424]
[0425] Among them, e i is the attention score of the i-th task, which is optimized by minimizing the following weighted loss function:
[0426]
[0427] S78. Implement an intelligent decision support system, make real-time decisions through a multi-task counterfactual reinforcement learning model, and define the final decision function D as:
[0428] D = argmax a π(s,a);
[0429] Where s is the current state, a is the optional action, and π(s,a) is the action probability output by the policy function;
[0430] S79. Introduce high-order counterfactual reasoning to generate counterfactual samples at multiple levels:
[0431] x( k+1 )=x( k )+δ( k )
[0432] Among them, δ( k ) is the perturbation amount of the kth layer, and the multi-level loss function is:
[0433]
[0434] S710, introduce complex reinforcement learning mechanism combined with Gaussian process to define the strategy function:
[0435] π(s t )=GP(s t ,θ);
[0436] Among them, GP is the Gaussian process model, θ is the model parameter, and optimization is performed by minimizing the following loss function:
[0437]
[0438] S711. Introducing complex regularization terms in the optimization process:
[0439]
[0440] in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature;
[0441] S712, adaptive moment estimation using adaptive optimization algorithm:
[0442]
[0443] Among them, η is the learning rate, m t is the first-order moment estimate of the gradient, v tis the second moment estimate of the gradient.
[0444] In this implementation, S8 includes the following steps:
[0445] S81. Construct a heterogeneous information network and define a graph structure G = (V, E) containing different types of nodes and edges, where the node set V and the edge set E are expressed as:
[0446] V={v1,v2,…,v n};
[0447] E={e1,e2,…,e m};
[0448] Among them, v i are of different types T v , edge i are different relationship types T e ;
[0449] S82. Construct a meta-path to represent different types of relationships in a heterogeneous information network, and define the meta-path P as:
[0450] P=(A1,R1,A2,R2,…,R k-1 ,A k );
[0451] Among them, A i Indicates the node type, R i Indicates the relationship type;
[0452] S83. Perform a random walk on each meta-path P to generate sequence data S:
[0453] S=(v1,v2,…,v l );
[0454] Among them, v i is a node, and the sequence length is l;
[0455] S84. Map the nodes in the heterogeneous information network to a low-dimensional vector space through a multimodal embedding method; define the embedding matrix M as:
[0456] M = MM-HIN2Vec(G);
[0457] Among them, MM-HIN2Vec is an embedding method for multimodal heterogeneous information networks, and G is a graph structure;
[0458] S85. Use graph convolutional variational autoencoder to extract features from the embedded low-dimensional vector and define the node feature matrix H as:
[0459] H = GCVAE(M);
[0460] Among them, GCVAE is the graph convolutional variational autoencoder model, and M is the embedding matrix;
[0461] S86. Use graph generation adversarial network to generate and train features, and define the loss functions of the generator and discriminator as:
[0462]
[0463] Among them, G is the generator, D is the discriminator, and p z (z) is the noise distribution, p data (x) is the data distribution;
[0464] S87, introduce complex regularization terms:
[0465]
[0466] in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature;
[0467] S88, visualization technology, visualizes the embedded low-dimensional vector and classification clustering results:
[0468] V = DL-Vis(H);
[0469] Among them, DL-Vis is a visualization algorithm based on deep learning, which is used to map high-dimensional data into two-dimensional space for visualization;
[0470] S89. Generate analysis reports. Combine visualization results and model outputs to generate reports containing data insights and decision recommendations:
[0471]
[0472] in, is the predicted label set, and C is the clustering result set.
[0473] Embodiment 1:
[0474] Assume that a traffic management department of a city hopes to improve the efficiency of urban traffic management through an intelligent data collection and analysis system, and realize real-time monitoring and prediction of traffic flow, so as to optimize traffic signal control and reduce traffic congestion. In order to achieve this goal, the traffic management department decides to use the method of the present invention for data collection and analysis.
[0475] In this project, the traffic management department deployed a large number of sensors throughout the city, including road traffic flow sensors, traffic cameras and environmental sensors. The data collected by the sensors include the number of vehicles, vehicle speed, license plate information, weather conditions, etc. The data is transmitted to the central database in real time through the API interface. In addition, historical traffic data and traffic information data provided by third parties are integrated to form a comprehensive multi-source data set.
[0476] To ensure data quality and consistency, the system uses synthetic control methods to create a virtual control group that simulates baseline traffic flow data under undisturbed conditions. This approach can effectively identify and correct abnormal data points, improving data accuracy and reliability.
[0477] In the data preprocessing stage, the system uses immune algorithms for anomaly detection and correction. The algorithm constructs an antigen data set and antibody library, calculates the affinity between data points, and generates candidate correction data points through a clonal selection algorithm, ultimately achieving effective correction of abnormal data. Multi-scale convolutional networks are used to denoise and standardize data, thereby extracting multi-level features to ensure data cleanliness and consistency.
[0478] In the feature extraction and analysis stage, the system uses topological data analysis and persistent homology methods to extract persistent features from traffic data and identify high-dimensional topological structures. Combining diffusion mapping and manifold learning techniques, nonlinear dimensionality reduction and analysis are performed on high-dimensional features to capture complex structures and patterns in the data and provide more accurate traffic flow predictions.
[0479] In order to construct complex traffic relationship diagrams, the system uses graph neural networks and graph embedding methods to model and analyze high-order relationships. In addition, non-negative matrix decomposition and tensor decomposition methods are used to collaboratively train and integrate traffic data from different perspectives to improve the depth and breadth of data analysis.
[0480] In terms of model adaptive adjustment and decision support, the system adopts counterfactual reasoning method and combines reinforcement learning to achieve intelligent decision support. The system can adjust the model in real time according to the new data environment, provide optimized traffic signal control strategy, and meet the needs of real-time data processing and analysis.
[0481] The system also provides high-dimensional data visualization tools through heterogeneous information network embedding methods, generates detailed analysis reports and intelligent decision-making recommendations, helps traffic management personnel intuitively understand and apply analysis results, and improves user experience.
[0482] The project was put into trial operation in a central area of a city from January 1, 2023 to June 30, 2023. The following are some specific data during the trial operation, which prove the beneficial effects of the present invention.
[0483] Table 1 Some specific data during operation
[0484] project data Traffic flow data collection 5000 data per second Data preprocessing time 100,000 data items per minute Abnormal data correction rate 98% Peak hour forecast accuracy 95% Traffic congestion duration reduced 20% Signal optimized response time 5 seconds Reduced vehicle waiting time 15% Customer satisfaction 85%
[0485] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. Intelligent big data collection and analysis method and system based on deep learning, characterized in that: The steps include: S1. Obtain data by configuring sensors, databases, API interfaces and IoT devices; S2, use immune algorithm for anomaly detection and correction, and use multi-scale convolutional network for data denoising and standardization; S3, using topological data analysis and persistent homology methods to extract persistent features from the data and identify high-dimensional topological structures in the data; S4, using diffusion mapping and manifold learning techniques to perform nonlinear dimensionality reduction and analysis on the extracted high-dimensional features to capture the complex structures and patterns in the data; S5. Use graph neural networks and graph embedding methods to build complex relationship graphs in data and model and analyze high-order relationships; S6. Use non-negative matrix decomposition and tensor decomposition methods to collaboratively train data from different perspectives and integrate multi-perspective information; S7. Use counterfactual reasoning methods to achieve adaptive adjustment of the model in new data environments, and combine reinforcement learning for intelligent decision support; S8. Utilize heterogeneous information network embedding methods to provide visualization tools for high-dimensional data and generate analysis reports and intelligent decision-making recommendations.
2. The deep learning-based intelligent big data collection and analysis method and system according to claim 1, characterized in that: The S1 comprises the following steps: S11. Configure sensors, databases, API interfaces and IoT devices to build a multi-source data collection system; S12. Extract data from different databases through API interfaces and integrate information from various databases; S13. Use IoT devices to collect sensor data in real time and collect information such as environment and device status; S14. Integrate multi-source data and build a comprehensive dataset.
3. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S2 comprises the following steps: S21. Use immune algorithm for anomaly detection and correction: Define the antigen data set A = {a1, a2, ..., a n } and antibody library B = {b1, b2, …, b m }, where a i represents the i-th data point, b j represents the jth antibody; Calculate the affinity matrix AFF: Among them, ||a i -b j || represents data point a i and antibody b j The Euclidean distance between them, σ is the scale parameter of the affinity function; Screen high-affinity antibodies based on the affinity matrix and generate candidate correction data points through the clone selection algorithm: Among them, A′ i is the corrected data point; S22, use multi-scale convolutional network to denoise data, using multi-scale convolution kernel K s Perform multi-level feature extraction on data X: F s =K s *X+b s ; Among them, * represents the convolution operation, K s represents the convolution kernel of scale s, b s is the bias term, F s is the feature map at scale s; Through the adaptive feature fusion strategy, the multi-scale feature map F s Perform weighted fusion to generate denoised data X′: Among them, α s Represents the feature weight coefficient of scale s calculated by the attention mechanism: Among them, β s is the feature importance score of scale s; S23, standardize the denoised data and calculate the mean μ and standard deviation μ of the denoised data: Where n is the number of data points, X′ i represents the i-th data point after denoising; S24. Standardize the data: Among them, X″ i is the i-th data point after standardization.
4. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S3 comprises the following steps: S31. Construct the topological space of data and transform the data X″ i Represented as a point cloud P = {x1, x2, ..., x n }; S32. Use random geometric complexes to perform topological construction on the point cloud P and generate random geometric complexes R(r): Among them, r is the connection radius, ||x i -x j || represents the data point x i and x j The Euclidean distance between S33. Calculate the persistent homology group under different connection radius r, and obtain the homology group H under each parameter k (R(r)): Where k represents the dimension of homology, is the boundary operator; S34. Use topology optimization entropy to optimize the persistent strips and record the birth and death relationship of topological features under different parameters ∈: Barcode={(b i ,d i )∣0≤b i <d i ≤∞}; Among them, b i is the birth parameter of topological features, d i is the extinction parameter of the topological feature; the topology optimization entropy is defined as follows: Among them, σ is the probability density of the i-th point, and λ is the regularization parameter; S35. Convert the persistent strip into a persistent graph to record the persistence of topological features: Diagram={(b i ,d i )∣0≤b i <d i ≤∞}; Among them, each point)b i ,d i ) represents the persistence of a topological feature; S36. Use high-dimensional homology convolutional neural networks to learn persistence graphs and extract deep representations of topological features: z = HD-HCNN (Diagram); Among them, z is the extracted deep topological feature vector, HD-HCNN represents the high-dimensional homology convolutional neural network, and the high-dimensional homology convolutional neural network uses the following convolution operation: Among them, h (l) represents the features of the lth layer, represents the weight matrix of the lth layer at the sth scale, A s is the adjacency matrix under scale s, b (l) is the bias, σ is the nonlinear activation function; S37. Generate new topological structures using generative adversarial networks: G′=GAN(z); Among them, G′ is the generated graph structure, and GAN represents the generative adversarial network; S38. Perform feature optimization and dimensionality reduction on the generated graph structure, and use high-dimensional space nonlinear projection technology to map high-dimensional topological features to low-dimensional space: e=tanh(W p G′+b p ); Among them, W p is the projection weight matrix, b p is the projection bias, tanh is the activation function; S39. Use low-dimensional topological feature vector e for data classification and clustering: Class=softmax(W c e b c ); Among them, W c is the classification weight matrix, b c is the classification bias vector.
5. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S4 comprises the following steps: S41, construct similarity matrix W, define data point x i and x j The similarity between ij : Among them, ||x i -x j || represents the data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter; S42, construct an angle matrix D, whose element d ii is the row sum of the similarity matrix W: S43. Calculate the normalized Laplace matrix L: L=I-D -1 / 2 WD -1 / 2 ; Where I is the identity matrix; S44, perform eigenvalue decomposition to solve the eigenvalues and eigenvectors of the normalized Laplace matrix L: Lφ k =λ k f k ; Among them, λ k and φ k are the kth eigenvalue and eigenvector respectively; S45. Use the eigenvectors corresponding to the first m smallest non-zero eigenvalues to construct a dimensionality reduction mapping: y i =(φ1(x i ),φ2(x i ),…,φ m (x i )); Among them, y i is the data point after dimensionality reduction; S46. Introducing high-dimensional nonlinear metrics D NL : Among them, F t (x i ) represents the data point x at time step t i Nonlinear transformation of : F t ( x i )=ReLU(W t y i +b t ): Among them, W t and b t are the weight matrix and bias at time step t, respectively, and ReLU is the activation function; S47. Use variational graph neural network to learn and optimize the features after dimensionality reduction: q(z∣y)=N(z:µ(y),σ 2 (y))? Among them, μ and σ are the mean and variance functions in VGNN, defined as: μ(y)=W μ y+b μ ; σ(y)=exp(W σ y+b σ ); Among them, W μ , W σ is the weight matrix, b μ , b σ is bias; S48. Optimize the feature representation after dimensionality reduction through variational inference: z i =μ(y i )+σ(y i )⊙∈; Among them, ∈ is a standard normal distribution random variable, ⊙ represents the element-by-element product; S49. Use the multi-level high-order spectral clustering method to cluster the final features, and define the optimization objective function as: Among them, L is the number of clustering layers, K is the number of clusters in each layer, is the kth cluster in the lth layer, For data point x i Belongs to cluster The probability of , β is the regularization parameter, n (l) is the total number of data points in layer l.
6. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S5 comprises the following steps: S51. Build a data relationship graph G, which contains a node set V and an edge set E. Define node v i and v j The edge weight w between ij for: Among them, ||x i -x j || represents the data point x i and x j The Euclidean distance between them, σ is the similarity scale parameter; S52, construct an adjacency matrix A, whose element a ij Represents node v i and v j The connection relationship between: S53, use Graph Convolutional Network (GCN) to extract features from graph structure data and define node features H (0) For input data X, the node features are updated by the following iterations: in, is the adjacency matrix with self-loops added, for The degree matrix, W (l) is the weight matrix of the lth layer, σ is the activation function, H (l) is the node feature of the lth layer; S54. Introduce a self-supervised learning mechanism to further optimize graph embedding through a contrast loss function. Define the contrast loss as: Among them, h i and are the feature vectors of the positive sample pairs, is the feature vector of the negative sample pair, sim(·) is the similarity measurement function; S55. Use graph embedding technology to map high-dimensional graph features to low-dimensional space: Z=GraphEmbed(H (L) ); Among them, H (L) is the last layer output of GCN, Z is the low-dimensional embedding representation, and GraphEmbed is the graph embedding function; S56. Use graph variational autoencoder to further optimize graph embedding representation and define variational loss function as: Where q(z|x) is the approximate posterior distribution, p(z) is the prior distribution, and KL(·||·) is the KL divergence; S57. Introduce a multi-view graph convolutional network to capture the node relationship under different viewpoints. The specific definition is as follows: in, is the adjacency matrix of the vth view, for The degree matrix, α v is the weight coefficient of the vth viewing angle, is the weight matrix of the vth view in the lth layer; S58. Finally, the output features of the multi-view graph convolutional network are used to perform relationship modeling and analysis to obtain a high-dimensional representation of the nodes: Z (MV) =H (L) ; Among them, Z (MV) It is the final output feature of the multi-view graph convolutional network; S59. Cluster the high-dimensional representation of the nodes through high-order spectral clustering, and define the optimization objective function as: Among them, K is the number of clusters, C k is the kth cluster, p(v i ∣C k ) is the node v i Belongs to cluster C k The probability of , γ is the regularization parameter, and n is the total number of nodes.
7. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S6 comprises the following steps: S61, constructing a data matrix X and a multi-view tensor T, where X represents a basic feature matrix of the data, and T represents a multi-view tensor; S62. Perform non-negative matrix decomposition to decompose the data matrix X into two non-negative matrices W and H, which are defined as follows: X≈WH: Among them, W ≥ 0 is the basis matrix, H ≥ 0 is the coefficient matrix; optimization is performed by minimizing the following loss function: Among them, ||·|| F is the Frobenius norm, λ is the regularization parameter; S63. Construct a multi-view tensor T, whose elements T ijk Indicates the relationship between the jth feature and the kth feature at the i-th perspective; S64, perform high-order tensor decomposition to decompose the multi-view tensor T into a set of factor matrices {A i }, defined as follows: Among them, λ r is the weight coefficient, A ir is the rth factor matrix, represents the tensor product; it is optimized by minimizing the following loss function: S65. Introduce a collaborative training mechanism, integrate multi-view information by jointly optimizing non-negative matrix decomposition and high-order tensor decomposition, and define the joint loss function as: I Joint =L NMF +αL HOTD ; Among them, α is the weight parameter of collaborative training; S66. Use variational Bayesian inference to optimize the joint model and define the variational posterior distribution q(θ) as: Where θ represents the model parameters and φi is the variational parameter; the optimization is performed by minimizing the following evidence lower bound: L VBI =E q(θ) [log p(X,T|θ)]-KL(q(θ)||p(θ)); Where KL(·||·) is the KL divergence and p(θ) is the prior distribution; S67. Adaptive sparse regularization is introduced in variational Bayesian inference, and the adaptive sparse regularization term is defined as: Among them, η ij is the adaptive sparse parameter, ∈ is a small value to prevent division by zero; S68, perform multi-view information fusion and use the fused feature matrix \(Z\) for data analysis: Among them, Z is the fused feature representation; S69. Perform cluster analysis on the fused features through high-order spectral clustering, and define the clustering objective function as: Among them, K is the number of clusters, C k is the kth cluster, p(z i |C k ) is the feature z i Belongs to cluster C k The probability of , β is the regularization parameter, n is the total number of features, a ij is the similarity matrix element, and γ is the similarity regularization parameter.
8. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S7 comprises the following steps: S71. Construct a counterfactual reasoning model and define the counterfactual sample generation process. Given input data x and target output y, generate counterfactual samples x′ by perturbing the original input: x′=x+δ; Among them, δ is the perturbation amount, which is optimized by minimizing the following loss function: L CF =||f(x)-y|| 2 +λ||δ|| 2 ; Where f(x) is the predicted output of the model and λ is the regularization parameter; S72. Introduce an adaptive adjustment mechanism into the counterfactual reasoning model to achieve adaptive adjustment of the model in the new data environment by minimizing the difference between the counterfactual samples and the original samples: L Adj =L CF +α||x′-x|| 2 ; Among them, α is the adaptive adjustment weight parameter; S73. Combine the reinforcement learning (RL) method to build a policy gradient-based reinforcement learning model; define the state s, action a and reward r as: s t =x′ t ,a t =π(s t ),r t =R(s t ,a t ); Among them, π(s t ) is the policy function, R(s t ,a t ) is the reward function; S74. Optimize the policy function π using the policy gradient method by minimizing the following loss function: Among them, γ is the discount factor; S75. Introduce counterfactual reasoning results into the reinforcement learning model, optimize the strategy function through the adaptively adjusted counterfactual samples, and define the optimization goal as: L CRL =L PG +βL Adj ; Among them, β is the counterfactual reasoning weight parameter; S76. Combine the multi-task learning (MTL) method to build a multi-task counterfactual reinforcement learning model; define the multi-task loss function as: Where M is the number of tasks, is the counterfactual reinforcement learning loss for the i-th task; S77. Define attention weight α i for: Among them, e i is the attention score of the i-th task, which is optimized by minimizing the following weighted loss function: S78. Implement an intelligent decision support system, make real-time decisions through a multi-task counterfactual reinforcement learning model, and define the final decision function D as: D=argmax a π(s,a); Where s is the current state, a is the optional action, and π(s,a) is the action probability output by the policy function; S79. Introduce high-order counterfactual reasoning to generate counterfactual samples at multiple levels: x (k+1) =x (k) +δ (k) Among them, δ (k) is the perturbation amount of the kth layer, and the multi-level loss function is: S710, introduce complex reinforcement learning mechanism combined with Gaussian process to define the strategy function: π(s t )=GP(s t ,i); Among them, GP is the Gaussian process model, θ is the model parameter, and optimization is performed by minimizing the following loss function: S711. Introducing complex regularization terms in the optimization process: in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature; S712, adaptive moment estimation using adaptive optimization algorithm: Among them, η is the learning rate, m t is the first-order moment estimate of the gradient, v t is the second moment estimate of the gradient.
9. The method and system for intelligent big data collection and analysis based on deep learning according to claim 1, characterized in that: The S8 comprises the following steps: S81. Construct a heterogeneous information network and define a graph structure G = (V, E) containing different types of nodes and edges, where the node set V and the edge set E are expressed as: V={v1,v2,...,v n }; E={e1,e2,…,e m }; Among them, v i are of different types T v , edge i are different relationship types T e ; S82. Construct a meta-path to represent different types of relationships in a heterogeneous information network, and define the meta-path P as: P=(A1,R1,A2,R2,…,R k-1 ,A k ); Among them, A i Indicates the node type, R i Indicates the relationship type; S83. Perform a random walk on each meta-path P to generate sequence data S: S=(v1,v2,…,v l ); Among them, v i is a node, and the sequence length is l; S84. Map the nodes in the heterogeneous information network to a low-dimensional vector space through a multimodal embedding method; define the embedding matrix M as: M = MM-HIN2Vec(G); Among them, MM-HIN2Vec is an embedding method for multimodal heterogeneous information networks, and G is a graph structure; S85. Use graph convolutional variational autoencoder to extract features from the embedded low-dimensional vector and define the node feature matrix H as: H = GCVAE(M); Among them, GCVAE is the graph convolutional variational autoencoder model, and M is the embedding matrix; S86. Use graph generation adversarial network to generate and train features, and define the loss functions of the generator and discriminator as: Among them, G is the generator, D is the discriminator, and p z (z) is the noise distribution, p data (x) is the data distribution; S87, introduce complex regularization terms: in, is the expected value, ∈ is a small value to prevent division by zero, and η ij is the adaptive sparse parameter corresponding to the i-th data point and the j-th feature; S88, visualization technology, visualizes the embedded low-dimensional vector and classification clustering results: V = DL-Vis(H); Among them, DL-Vis is a visualization algorithm based on deep learning, which is used to map high-dimensional data into two-dimensional space for visualization; S89. Generate analysis reports. Combine visualization results and model outputs to generate reports containing data insights and decision recommendations: in, is the predicted label set, and C is the clustering result set.
Citation Information
Cited By
Graph neural network-based power grid dispatching decision-making method and large model
CN119294872A
Arterial aneurysm 3D point cloud automatic segmentation method combined with geometric topology analysis
CN119379721A
Converter station energy efficiency prediction system based on deep learning
CN119578664A
Deep learning-based rapid ore sorting method under adaptive illumination intensity
CN119863665A
Index characterization evaluation method based on aerospace equipment model
CN119886547A
