Dynamic data quality monitoring and repairing method based on adaptive algorithm

Abstract

The invention discloses a dynamic data quality monitoring and repairing method based on an adaptive algorithm, and the method comprises the steps: S1, carrying out the segmentation and slicing of data according to a time sequence, and generating an initial data graph structure; s2, dynamically adjusting a topological relation between nodes and edges of the graph by adopting a self-adaptive algorithm, and generating a dynamic graph structure; s3, generating a high-dimensional feature matrix through joint modeling of the initial time feature matrix and the initial space feature matrix; s4, fusing the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix to generate a multi-modal fusion feature matrix; s5, inputting the multi-modal fusion feature matrix into a generative adversarial network to generate a preliminary repair data matrix; s6, optimizing the preliminary repair data matrix through a microelement loss mechanism to generate a repair data matrix; and S7, verifying the repaired data matrix, and generating a verification report. The method has the advantages of high data recovery smoothness consistency and good verification and optimization real-time performance.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a dynamic data quality monitoring and repair method based on an adaptive algorithm. Background Art

[0002] With the rapid development of big data technology and artificial intelligence, the quality monitoring and repair of dynamic data have become increasingly important in various fields. Especially in real-time dynamic scenarios such as the Internet of Things, financial data analysis, and intelligent transportation, the stability and accuracy of data quality directly affect the reliability of the system and the scientific nature of decision-making. However, due to its complexity and spatio-temporal evolution characteristics, dynamic data is often vulnerable to the interference of missing values, outliers, and noise, posing challenges to high-quality data analysis and model applications.

[0003] In the prior art, the quality monitoring and repair of dynamic data usually rely on traditional static models and simple interpolation methods. These methods are inadequate when dealing with complex dynamic data. Specifically, the prior art has the following main defects:

[0004] 1. Lack of dynamic graph structure modeling ability: Traditional methods often rely on fixed static data structures and are unable to dynamically adjust the model structure to adapt to the real-time changes of data. This defect makes traditional methods insufficient in capturing the spatio-temporal evolution characteristics of dynamic data.

[0005] 2. Insufficient support for spatio-temporal feature fusion: In the prior art, time series analysis and spatial feature modeling are usually carried out independently, lacking the ability to unify the modeling of time features and spatial features, and unable to comprehensively reflect the multi-dimensional characteristics and complex associations of data.

[0006] 3. Single data repair strategy: Data repair in the prior art mostly adopts simple interpolation or filling strategies, failing to fully consider the global relevance and dynamic change characteristics of data, resulting in the lack of smoothness and consistency in the repair results.

[0007] 4. Insufficient dynamic support for verification and optimization: Traditional data verification methods are mostly static checks, lacking an effective feedback mechanism for the real-time dynamic changes of data, and unable to dynamically optimize data models and parameters according to verification results.

[0008] 5. Weak ability to process multi-modal data: In dynamic data scenarios, data from different sources and types has multi-modal characteristics, but the prior art lacks effective fusion and unified coding methods for multi-modal features, and is unable to fully explore the internal associations of data.

[0009] Therefore, how to provide a dynamic data quality monitoring and repair method based on an adaptive algorithm is an urgent problem to be solved by those skilled in the art. Summary of the Invention

[0010] An object of the present invention is to propose a dynamic data quality monitoring and repair method based on an adaptive algorithm. The present invention fully combines an adaptive dynamic graph structure generation technology, a spatio-temporal aware graph neural network, multi-modal feature fusion analysis, and a generative adversarial network to optimize the repair strategy, and details the implementation process of dynamic data quality monitoring and repair, having the advantages of strong dynamic modeling ability, accurate spatio-temporal correlation capture, high smooth consistency of data repair, and good real-time performance of verification and optimization.

[0011] A dynamic data quality monitoring and repair method based on an adaptive algorithm according to an embodiment of the present invention includes the following steps:

[0012] S1. Obtain a data stream in a dynamic environment, segment and slice the data stream according to a time series, and extract an initial time feature matrix and an initial space feature matrix to generate an initial data graph structure;

[0013] S2. According to the initial data graph structure and real-time data changes, use an adaptive algorithm to dynamically adjust the topological relationship of the nodes and edges of the graph to generate a dynamic graph structure that can reflect the real-time dynamics of the data;

[0014] S3. Use the dynamic graph structure to construct a spatio-temporal aware graph neural network model, and generate a high-dimensional feature matrix containing node time-series evolution features and spatial correlation features by jointly modeling the initial time feature matrix and the initial space feature matrix;

[0015] S4. Fuse the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix, and generate a multi-modal fusion feature matrix through unified coding by multi-modal feature analysis;

[0016] S5. Input the multi-modal fusion feature matrix into a generative adversarial network, train a generation model to predict missing values and outliers in the data, and at the same time train a discriminant model to evaluate the authenticity of the generated data to generate a preliminary repaired data matrix;

[0017] S6. Optimize the preliminary repaired data matrix through a differentiable loss mechanism to generate a smooth and spatio-temporally consistent repaired data matrix;

[0018] S7. Verify the repaired data matrix, including spatio-temporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the parameters of the dynamic graph structure and the spatio-temporal aware graph neural network model according to the verification report.

[0019] Optionally, the S2 specifically includes:

[0020] S21. Construct an initial data graph structure , where is the initial node set, representing the data entity of the data flow in the dynamic environment, is the initial edge set, which represents the relationship between nodes, and for each node Assign initial eigenvector and each edge Assign initial weight ;

[0021] S22, calculate each node based on real-time collected dynamic data The node feature change : ;

[0022] in, For Node In time The characteristic value of is the eigenvalue at the previous moment, To prevent the denominator from being zero;

[0023] Calculate each edge The weight change : ;

[0024] in, Represents two nodes connected by an edge and The weight change of and Node and In time The characteristic value of represents the modulus of the eigenvalue, To avoid constants with zero denominators;

[0025] According to the node feature change and weight change , dynamically adjust the initial weight of each edge : ;

[0026] in, is the initial weight of the edge, is the adjusted edge weight, and are the dynamic adjustment factors for weight change and node feature change, respectively. and Node and The node feature change amount;

[0027] S23. Perform threshold judgment on the adjusted edge weights, and set a dynamic threshold based on the average value and standard deviation of the edge weights : ; ;

[0028] Among them, represents the number of edges in the initial edge set , is the adjusted edge weight, and are preset adjustment coefficients. For each edge , compare the adjusted edge weight with the dynamic threshold . When the condition is satisfied, retain the edge in the updated edge set ;

[0029] S24. According to the updated edge set , combined with the node feature change amount , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure , where and represent the adjusted node set and the updated edge set respectively;

[0030] S25. Perform timeliness verification on the dynamic graph structure , including whether the update frequency of nodes is synchronized with the dynamic data change and whether the connectivity of the edge set meets the modeling requirements;

[0031] S26. During the dynamic adjustment process, record the nodes and edges that do not meet the conditions to generate an adjustment log.

[0032] Optionally, the specific content of S3 includes:

[0033] S31. Use the dynamic graph structure to construct a spatio-temporal aware graph neural network model, and use the time feature and the spatial adjacency matrix as inputs. Among them, is the node set of the dynamic graph, is the edge set of the dynamic graph. Each node is given an initial feature vector , and the time feature is generated by the dynamic attribute changes of the node at different time steps. The spatial adjacency matrix is composed of the edge set Definition;

[0034] S32. Generate the spatio-temporal embedding representation of the node through the joint modeling of the node's own features, temporal features, and neighbor node features;

[0035] S33. Aggregate the spatio-temporal embedding representations of all nodes to generate a high-dimensional spatio-temporal feature set of the dynamic graph nodes;

[0036] S34. Construct a multi-layer spatio-temporal aware graph neural network, define a multi-layer recursive update rule, and calculate the recursive feature representation of the node through the message passing mechanism. The update rule for the th layer is: ;

[0037] where, is a non-linear activation function, is the trainable weight matrix for the th layer, represents the normalized aggregation of neighbor node features, is the feature representation of node in the th layer of the graph neural network, is the feature representation of node in the th layer of the graph neural network;

[0038] S35. Through multi-layer iterative updates, generate a high-dimensional feature matrix representing the final spatio-temporal embedding features of all nodes after multi-layer recursion for the dynamic graph structure .

[0039] Optionally, the specific steps of S4 include:

[0040] S41. Define a feature fusion rule, and splice the high-dimensional feature matrix , the initial temporal feature matrix and the initial spatial feature matrix according to the node index to generate an initial multi-modal feature matrix : ;

[0041] where, represents the feature splicing operation of the matrix, is a multi-modal feature matrix containing spatio-temporal features, temporal features, and spatial features;

[0042] S42. Normalize the initial multi-modal feature matrix to generate a normalized feature matrix ;

[0043] S43. Use an autoencoder to uniformly encode the normalized feature matrix and construct an autoencoder model. Its encoding rule is: ;

[0044] Among them, is the multi-modal fusion feature matrix after uniform encoding, is the activation function of the encoder, and are the weight matrix and bias vector of the encoder respectively.

[0045] Optionally, the specific steps of S5 are as follows:

[0046] S51. Input the multi-modal fusion feature matrix into the generative adversarial network, where the generative adversarial network includes a generative model and a discriminative model ;

[0047] S52. Define the prediction rule of the generative model . The generative model takes the multi-modal fusion feature matrix as input and generates a preliminary repaired data matrix through a deep neural network: ;

[0048] Among them, is the activation function of the generative model, and are the weight matrix and bias vector of the generative model respectively;

[0049] S53. Define the discrimination rule of the discriminative model . The discriminative model takes the preliminary repaired data matrix and the real data matrix as input, where the real data matrix is composed of the data stream in the dynamic environment, and calculates its authenticity score and through a binary classification neural network: ;

[0050] Among them, is the activation function of the discriminative model, and are the weight matrix and bias vector of the discriminative model respectively, represents the concatenated input of real data and generated data;

[0051] S54. Conduct adversarial training on the generative model and the discriminative model: ;

[0052] Among them, is the adversarial loss function, , are weight coefficients, , are bias terms, is the balance coefficient, represents the square of the Euclidean norm, represents the mean squared error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data, and respectively represent the real data and the generated data 's probability scores of being judged as real or fake, , and respectively represent the expectations on the real data, the generated data, and their joint distribution, is the adjustment coefficient, represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generation model and the discriminant model, the generation model can generate repaired data close to the real data distribution;

[0053] S55. Output the preliminary repaired data matrix after training the generative adversarial network .

[0054] Optionally, the S6 specifically includes:

[0055] S61. Based on the preliminary repaired data matrix , construct a spatio-temporal consistency optimization model, and input the preliminary repaired data matrix , the initial time feature matrix and the initial spatial feature matrix ;

[0056] S62. Define the spatio-temporal smoothness constraint, and smooth the repaired data in the preliminary repaired data matrix in the time domain and the spatial domain respectively: ;

[0057] Among them, is the spatio-temporal smoothness loss function, is the total number of time steps, and are balance coefficients, represents the mean squared error between the repaired data and the repaired data at the previous moment, is the node and The connection weights and respectively represent the repair data of nodes and ;

[0058] S63. Define the comprehensive optimization objective function , combining spatio-temporal consistency constraints and data repair errors: ;

[0059] Among them, is the balance coefficient of the error constraint, represents the mean square error between the repair data and the true data;

[0060] S64. Use the gradient descent method to solve the comprehensive optimization objective function to obtain the repair data matrix : ;

[0061] Among them, is the learning rate, represents the gradient of the loss function with respect to the repair data matrix.

[0062] Optionally, the S7 specifically includes:

[0063] S71. Input the optimized repair data matrix , the initial time feature matrix and the initial spatial feature matrix to construct a verification model for spatio-temporal correlation consistency checking;

[0064] S72. Define the time consistency verification rule and calculate the consistency index of the repair data matrix in the time domain: ;

[0065] Among them, is norm, is the time consistency adjustment parameter, is the total number of time steps, represents the repair data matrix at the previous moment;

[0066] S73. Define the spatial consistency verification rule and calculate the consistency index of the repair data matrix in the spatial domain: ;

[0067] Among them, is the connection weight between nodes in the adjacency matrix and , is the spatial consistency adjustment parameter represents the repair data matrix of node , represents the repair data matrix of node ;

[0068] S74. Combine the time consistency and spatial consistency verification rules to construct a comprehensive verification index : ;

[0069] wherein and are the weight parameters of time and spatial consistency respectively, used to dynamically balance the importance of different verification indexes;

[0070] S75. Generate a verification report according to the calculation result of the comprehensive verification index . The report content includes spatio-temporal consistency analysis and overall quality analysis of the repair data;

[0071] S76. Based on the verification report, dynamically update the topological structure of the dynamic graph structure and the parameters of the spatio-temporal aware graph neural network model.

[0072] The beneficial effects of the present invention are as follows:

[0073] (1) By combining the generation of adaptive dynamic graph structure, spatio-temporal aware graph neural network, multi-modal feature fusion analysis and generative adversarial network to optimize the repair strategy, the present invention provides a deep understanding of the quality of dynamic data and the ability of dynamic adaptation, enabling the system to capture the spatio-temporal correlation characteristics and their evolution laws of the data, so as to effectively cope with the problems of missing values, outliers and noise in dynamic data. By introducing a dynamically adjusted graph structure and an efficient spatio-temporal joint modeling technology, the system can accurately extract the global and local features of the data, providing comprehensive support for data quality monitoring.

[0074] (2) By constructing a dynamic data verification and optimization framework and using spatio-temporal consistency verification indexes combined with a comprehensive optimization objective function, the present invention realizes the comprehensive quality analysis and real-time optimization of the repair data. Combining the adversarial training mechanism of the generative adversarial network and the unified coding ability of the autoencoder, the system can not only efficiently repair the data, but also continuously optimize the model structure and parameter configuration during the repair process, thus greatly improving the accuracy and consistency of data repair.

[0075] (3) By comprehensively integrating spatio-temporal modeling, multi-modal feature analysis, and dynamic verification and optimization technologies, the present invention provides a means for multi-dimensional perspective monitoring and real-time repair of dynamic data quality. The system can automatically adapt to the dynamic changes of data, reduce the dependence on manual intervention, and at the same time can dynamically adjust the parameter configuration of the dynamic graph structure and spatio-temporal perception model, significantly improving the intelligence and adaptability of dynamic data quality monitoring and repair, and providing a solid technical guarantee for the wide application of dynamic data scenarios. Brief Description of the Drawings

[0076] The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention, and do not constitute a limitation to the present invention. In the accompanying drawings:

[0077] Figure 1 is the overall framework diagram of a method for monitoring and repairing dynamic data quality based on an adaptive algorithm proposed by the present invention. Detailed Embodiments

[0078] Now, the present invention will be further described in detail with reference to the accompanying drawings. These drawings are all simplified schematic diagrams, only illustrating the basic structure of the present invention in a schematic manner, so they only show the components related to the present invention.

[0079] Refer to Figure 1 , a method for monitoring and repairing dynamic data quality based on an adaptive algorithm, includes the following steps:

[0080] S1. Obtain the data stream in the dynamic environment, segment and slice the data stream according to the time series, and extract the initial time feature matrix and the initial space feature matrix to generate an initial data graph structure;

[0081] In this embodiment, S1 specifically includes:

[0082] Obtain the real-time data streams of multiple data sources in the dynamic environment, preprocess the data streams, segment and slice the preprocessed data streams in chronological order, each slice corresponding to a fixed time interval, the size of the time interval being set according to the requirements of the application scenario, slice the data stream into a plurality of continuous time period sets, extract the initial time feature matrix and the initial space feature matrix from the time slice data of the time period sets, and construct an initial data graph structure according to the extracted initial time feature matrix and initial space feature matrix.

[0083] S2. According to the initial data graph structure and the real-time data changes, use an adaptive algorithm to dynamically adjust the topological relationship of the nodes and edges of the graph to generate a dynamic graph structure that can reflect the real-time dynamics of the data;

[0084] In this embodiment, S2 specifically includes:

[0085] S21. Construct an initial data graph structure , where is the set of initial nodes, representing the data entities of the data flow in the dynamic environment, is the set of initial edges, representing the association relationships between nodes, and for each node assign an initial feature vector and for each edge assign an initial weight ;

[0086] S22. Calculate the node feature change amount of each node based on the dynamically collected data in real time : ;

[0087] where is the eigenvalue of node at time , is the eigenvalue of the previous moment, is a constant to prevent the denominator from being zero;

[0088] Calculate the weight change amount of each edge : ; ;

[0089] where represents the weight change amount of the two nodes and connected by the edge, and are the eigenvalues of nodes and at time respectively, represents the modulus length of the eigenvalue, is a constant to avoid the denominator from being zero;

[0090] Dynamically adjust the initial weight of each edge according to the node feature change amount and the weight change amount : ;

[0091] where is the initial weight of the edge, is the adjusted edge weight value, and are the dynamic adjustment factors of the weight change amount and the node feature change amount respectively, and are nodes respectively and the variation of node features;

[0092] S23. Perform a threshold judgment on the adjusted edge weights Based on the average and standard deviation of the edge weights, set a dynamic threshold : ;

[0093] Among them, represents the number of edges in the initial edge set ; is the adjusted edge weight, and are preset adjustment coefficients. For each edge , compare the adjusted edge weight with the dynamic threshold . When the condition is satisfied, retain the edge in the updated edge set ;

[0094] S24. According to the updated edge set , combined with the variation of node features , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure , where and represent the adjusted node set and the updated edge set respectively;

[0095] S25. Perform a timeliness verification on the dynamic graph structure , including whether the update frequency of nodes is synchronized with the dynamic data change and whether the connectivity of the edge set meets the modeling requirements;

[0096] S26. During the dynamic adjustment process, record the nodes and edges that do not meet the conditions to generate an adjustment log.

[0097] In this embodiment, the topological relationship of the nodes and edges of the graph is dynamically adjusted through an adaptive algorithm to generate a dynamic graph structure that can reflect the real-time dynamics of the data, achieving precise capture and real-time update of the characteristics of dynamic data. Combining the dynamic calculation rules of the node feature change amount and the edge weight change amount ensures the sensitivity and adaptability of the graph structure to data changes, thus comprehensively reflecting the spatio-temporal correlation relationship between the data. In addition, by intelligently adjusting the dynamic threshold and updating the edge set, the connectivity and topological consistency of the graph are further optimized, providing a high-quality input basis for subsequent spatio-temporal perception modeling, enhancing the flexibility and efficiency of dynamic graph modeling, and providing an efficient and real-time structured solution for the analysis and processing of complex dynamic data.

[0098] S3. Use the dynamic graph structure to construct a spatio-temporal perception graph neural network model. By jointly modeling the initial time feature matrix and the initial space feature matrix, a high-dimensional feature matrix containing the node time-series evolution features and space correlation features is generated;

[0099] In this embodiment, S3 specifically includes:

[0100] S31. Use the dynamic graph structure , construct a spatio-temporal perception graph neural network model, and combine the time features and the spatial adjacency matrix as inputs, where is the node set of the dynamic graph, is the edge set of the dynamic graph, and each node is given an initial feature vector . The time feature is generated by the dynamic attribute changes of the node at different time steps, and the spatial adjacency matrix is defined by the edge set ;

[0101] S32. Generate the spatio-temporal embedding representation of the node through the joint modeling of the node's own features, time features, and neighbor node features;

[0102] S33. Aggregate the spatio-temporal embedding representations of all nodes to generate a high-dimensional spatio-temporal feature set of the dynamic graph nodes;

[0103] S34. Construct a multi-layer spatio-temporal perception graph neural network, define multi-layer recursive update rules, and calculate the recursive feature representation of the node through the message passing mechanism. The update rule for the layer is: ;

[0104] where is a non-linear activation function, is the The trainable weight matrix of the layer, represents the normalized aggregation of neighbor node features, for node in the layer of the graph neural network, for node in the layer of the graph neural network;

[0105] S35. Through multi-layer iterative updates, generate a dynamic graph structure of the high-dimensional feature matrix representing the final spatio-temporal embedding features of all nodes after multi-layer recursion .

[0106] This embodiment constructs a spatio-temporal aware graph neural network model using a dynamic graph structure. By jointly modeling the initial time features and initial space features, it realizes the deep capture of the temporal evolution features and spatial correlation features of dynamic data. This model can not only perform spatio-temporal embedding representation on node features, but also effectively extract global spatio-temporal dependence relationships through multi-layer recursive update rules, thereby generating a high-dimensional feature matrix, comprehensively depicting the complex data relationships and dynamic evolution laws in the dynamic graph. In this way, the adaptability of the model in complex data environments is enhanced, providing accurate and efficient technical support for dynamic data quality monitoring and analysis.

[0107] S4. Fuse the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix, and generate a multi-modal fusion feature matrix through unified encoding by multi-modal feature analysis;

[0108] In this embodiment, S4 specifically includes:

[0109] S41. Define a feature fusion rule, and splice the high-dimensional feature matrix , the initial time feature matrix and the initial space feature matrix according to the node index to generate an initial multi-modal feature matrix : ;

[0110] wherein, represents the feature splicing operation of the matrix, is a multi-modal feature matrix containing spatio-temporal features, time features and space features;

[0111] S42. Perform normalization processing on the initial multi-modal feature matrix to generate a normalized feature matrix ;

[0112] S43. Use an autoencoder to process the normalized feature matrix Perform unified encoding and construct an autoencoder model. Its encoding rule is as follows: ;

[0113] Among them, is the multi-modal fusion feature matrix after unified encoding, is the activation function of the encoder, and are the weight matrix and bias vector of the encoder respectively.

[0114] In this embodiment, by fusing the high-dimensional feature matrix of the dynamic graph, the initial time feature matrix, and the initial space feature matrix, a unified multi-modal feature matrix is generated, and further unified encoding is performed on it using an autoencoder, realizing the efficient integration and feature extraction of multi-modal data. This method can not only capture the spatio-temporal correlation characteristics of the data, but also significantly reduce the redundant interference between feature dimensions through feature normalization and the non-linear mapping of the autoencoder, strengthening the multi-modal expression ability of the data, and thus laying a foundation for the subsequent repair and verification links of the generative adversarial network. This feature fusion method improves the accuracy and robustness of dynamic data processing, providing an efficient and flexible solution for dynamic data quality monitoring and repair.

[0115] S5. Input the multi-modal fusion feature matrix into the generative adversarial network, train the generative model to predict the missing values and outliers in the data, and at the same time train the discriminative model to evaluate the authenticity of the generated data, generating a preliminary repaired data matrix;

[0116] In this embodiment, S5 specifically includes:

[0117] S51. Input the multi-modal fusion feature matrix into the generative adversarial network, where the generative adversarial network includes a generative model and a discriminative model ;

[0118] S52. Define the prediction rule of the generative model . The generative model takes the multi-modal fusion feature matrix as input and generates a preliminary repaired data matrix through a deep neural network: ;

[0119] Among them, is the activation function of the generative model, and are the weight matrix and bias vector of the generative model respectively;

[0120] S53. Define the discriminative model Discrimination rules, the discrimination model uses the preliminary repaired data matrix and the real data matrix as inputs, where the real data matrix is composed of the data stream in the dynamic environment, and its authenticity score is calculated through a binary classification neural network and : ;

[0121] Among them, is the activation function of the discrimination model, and are the weight matrix and bias vector of the discrimination model respectively, represents the concatenated input of the real data and the generated data;

[0122] S54. Conduct adversarial training on the generation model and the discrimination model: ;

[0123] Among them, is the adversarial loss function, , are weight coefficients, , are bias terms, is the balance coefficient, represents the square of the Euclidean norm, represents the mean square error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data, and respectively represent the probability scores that the real data and the generated data are judged to be real or forged, , and respectively represent the expectations on the real data, the generated data, and their joint distribution, is the adjustment coefficient, represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generation model and the discrimination model, the generation model can generate repaired data close to the real data distribution;

[0124] S55. Output the preliminary repaired data matrix after the training of the generative adversarial network .

[0125] In this embodiment, the multi-modal fusion feature matrix is input into the generation model and the discriminant model through a generative adversarial network (GAN). The generation model predicts and repairs missing values and outliers in the data based on a deep neural network. The discriminant model evaluates the authenticity of the generated data and optimizes the generation result. Through adversarial training, it further approximates the real data distribution. At the same time, combined with a complex adversarial loss function, error optimization and adjacent node similarity constraints are performed on the repaired data, effectively ensuring the spatio-temporal consistency and smoothness of the repaired data. This method combines the powerful prediction ability of the generative adversarial network and the comprehensiveness of multi-modal feature fusion, improving the accuracy and stability of dynamic data repair, and providing an accurate and efficient solution for dynamic data quality monitoring and repair.

[0126] S6. Optimize the preliminary repaired data matrix through a differentiable loss mechanism to generate a smooth and spatio-temporally consistent repaired data matrix;

[0127] In this embodiment, S6 specifically includes:

[0128] S61. Based on the preliminary repaired data matrix , construct a spatio-temporal consistency optimization model, and input the preliminary repaired data matrix , the initial time feature matrix and the initial space feature matrix ;

[0129] S62. Define spatio-temporal smoothness constraints, and perform smoothing processing on the repaired data in the preliminary repaired data matrix in the time domain and the space domain respectively: ;

[0130] Among them, is the spatio-temporal smoothness loss function, is the total number of time steps, and are balance coefficients, represents the mean square error between the repaired data and the repaired data at the previous moment, is the connection weight between nodes and in the adjacency matrix, and respectively represent the repaired data of nodes and ;

[0131] S63. Define a comprehensive optimization objective function , combining spatio-temporal consistency constraints and data repair errors: ;

[0132] Among them, is the balance coefficient of the error constraint, Represents the mean square error between the repaired data and the true data;

[0133] S64. Use the gradient descent method to solve the comprehensive optimization objective function to obtain the repaired data matrix : ;

[0134] where, is the learning rate, represents the gradient of the loss function with respect to the repaired data matrix.

[0135] In this embodiment, by constructing a spatio-temporal consistency optimization model, jointly constraining the temporal domain smoothing and spatial domain correlation of the repaired data matrix, and using the comprehensive optimization objective function to iteratively optimize the repaired data, not only significantly improves the smoothness of the repaired data, but also effectively preserves the structural consistency of the data in the spatio-temporal dimension. By combining spatio-temporal consistency constraints and repair error optimization, the accuracy and logical consistency of the repaired data are enhanced, thereby ensuring that the optimized data can more truly reflect the dynamic evolution characteristics of the actual data, providing strong technical support for the high-quality repair of dynamic data.

[0136] S7. Verify the repaired data matrix, including spatio-temporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the dynamic graph structure and the parameters of the spatio-temporal aware graph neural network model according to the verification report.

[0137] In this embodiment, S7 specifically includes:

[0138] S71. Input the optimized repaired data matrix , the initial time feature matrix and the initial spatial feature matrix to construct a verification model for spatio-temporal correlation consistency check;

[0139] S72. Define the time consistency verification rule and calculate the consistency index of the repaired data matrix in the time domain: ;

[0140] where, is the norm, is the time consistency adjustment parameter, is the total number of time steps, represents the repaired data matrix at the previous moment;

[0141] S73. Define the spatial consistency verification rule and calculate the consistency index of the repaired data matrix in the spatial domain : ;

[0142] where is the connection weight between nodes and in the adjacency matrix, is the spatial consistency adjustment parameter, represents the repaired data matrix of node , represents the repaired data matrix of node ;

[0143] S74. Combine the time consistency and spatial consistency verification rules to construct a comprehensive verification index : ;

[0144] where and are the weight parameters of time and spatial consistency respectively, used to dynamically balance the importance of different verification indexes;

[0145] S75. Generate a verification report according to the calculation result of the comprehensive verification index , and the report content includes spatio-temporal consistency analysis and overall quality analysis of the repaired data;

[0146] S76. Dynamically update the topological structure of the dynamic graph structure and the parameters of the spatio-temporal aware graph neural network model based on the verification report.

[0147] In this embodiment, by constructing a spatio-temporal correlation consistency verification model, the optimized repaired data matrix is verified for consistency in the time domain and spatial domain. Using the time consistency and spatial consistency verification rules and combining the comprehensive verification index, a comprehensive analysis of the spatio-temporal characteristics of the repaired data is carried out. Through the time consistency verification, the smoothness and continuity of the data in the time evolution can be captured; through the spatial consistency verification, the coordination of the data in the spatial adjacency relationship can be identified. Finally, the weights of the time and space characteristics are dynamically balanced through the comprehensive verification index, a detailed verification report is generated, and the dynamic graph structure and model parameters are dynamically updated. This verification mechanism improves the overall consistency analysis ability of the repaired data, and at the same time realizes the real-time optimization of the dynamic graph and the spatio-temporal aware graph neural network model, providing a reliable verification and adjustment tool for the monitoring and repair of complex dynamic data quality.

[0148] Example 1:

[0149] To verify the feasibility of the present invention, the present invention is applied to the scenario of dynamic data quality monitoring and repair in an intelligent transportation system. The system monitors urban traffic data, including real-time road condition information, traffic flow statistics, vehicle speeds, and congestion conditions, etc. These data are the basis for urban traffic management and decision-making. However, due to diverse data sources and complex environments, missing values, outliers, and noise often occur, affecting the accuracy and efficiency of traffic management.

[0150] Recently, in the intelligent transportation management center of a certain city, system administrators discovered some problems in the traffic data, including long-term data missing in certain areas, inaccurate traffic flow statistics, and frequent abnormal peaks. To improve data quality and support more accurate traffic optimization decisions, the management center deployed the dynamic data quality monitoring and repair method based on an adaptive algorithm proposed by the present invention.

[0151] The intelligent transportation system first collects dynamic data from road surface cameras, vehicle networking sensors, and traffic management platforms. Through the adaptive algorithm of the present invention, the system constructs a dynamic graph structure to reflect the change trends and correlations of the data in real time. Using the spatio-temporal perception graph neural network A, the system jointly models the time series features and spatial characteristics in the traffic data, generating a high-dimensional feature matrix containing global correlations.

[0152] During traffic peak periods, for example, during the morning rush hour on a working day, the system conducts a fusion analysis of the multi-modal features of some key traffic nodes. Through the generative adversarial network, the system effectively repairs data missing and outliers. For example, in the traffic monitoring data between Dongzhimen and Sanlitun, the traffic flow statistics for some time periods are missing or abnormal. The system restores the missing values to a reasonable range through the repair algorithm and optimizes the abnormal peaks.

[0153] The system verifies the repaired data by constructing a spatio-temporal consistency verification model. In the time consistency verification, the smoothness of the repaired data is analyzed to ensure that the data changes conform to the real trends of traffic flow. In the spatial consistency verification, the traffic coordination of adjacent traffic nodes is evaluated to discover potential abnormal areas. According to the verification results, the system dynamically optimizes the dynamic graph structure and model parameters, further improving the accuracy and consistency of data repair. During the one-month test, the present invention significantly improved the quality of intelligent transportation data and the response ability of the system. The specific test data is shown in the following table:

[0154] Table 1 Intelligent Transportation Data Quality Optimization Test Report

[0155] As can be seen from the data table, the system before optimization performed poorly in key indicators such as data missing rate, outlier detection rate, and repair accuracy. In particular, the data missing rate was as high as 8.5%, seriously affecting the traffic management center's accurate grasp of the traffic flow on key sections. In addition, the response time was long and could not meet the requirements of real-time optimization. The optimized system, through dynamic modeling and real-time repair strategies, significantly reduced the data missing rate to 0.4%, while the outlier detection rate was increased to 97.8%, effectively making up for the deficiencies in data collection.

[0156] After optimization, the system, through the spatio-temporal consistency verification model, significantly improved the overall quality and coordination of data repair. The data consistency verification score increased from 67.5 before optimization to 95.3. This indicates that the repaired data is more logical in terms of time and space and conforms to the dynamic evolution law of the actual traffic flow. In addition, the accuracy of data repair increased from 83.4% to 98.7%, further verifying the remarkable effect of the present invention in dynamic data repair.

[0157] In actual tests, the present invention also significantly shortened the response time of data processing. The average time-consuming for repair and verification before optimization was 45 seconds, while after optimization it only took 8 seconds. The fast response ability enables the system to process large-scale dynamic data in real time, while improving the accuracy of traffic prediction, providing reliable support for urban traffic optimization and scheduling. These data fully illustrate the advantages of the present invention in dynamic data quality monitoring and repair.

[0158] The above are only the preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A dynamic data quality monitoring and repair method based on an adaptive algorithm, characterized in that, It includes the following steps: S1. Obtain the data stream in the dynamic environment, segment and slice the data stream according to the time series, and extract the initial time feature matrix and the initial spatial feature matrix to generate an initial data graph structure; S2. According to the initial data graph structure and the real-time data changes, adopt an adaptive algorithm to dynamically adjust the topological relationship of the nodes and edges of the graph, and generate a dynamic graph structure that can reflect the real-time dynamics of the data; S3. Use the dynamic graph structure to construct a spatio-temporal aware graph neural network model. By jointly modeling the initial time feature matrix and the initial spatial feature matrix, generate a high-dimensional feature matrix containing node time series evolution features and spatial correlation features; S4. Integrate the high-dimensional feature matrix with the initial time feature matrix and the initial spatial feature matrix, and perform unified encoding through multi-modal feature analysis to generate a multi-modal fusion feature matrix; S5. Input the multi-modal fusion feature matrix into a generative adversarial network, train the generative model to predict the missing values and outliers in the data, and at the same time train the discriminative model to evaluate the authenticity of the generated data, and generate a preliminary repaired data matrix; S6. Optimize the preliminary repaired data matrix through a differentiable loss mechanism to generate a smooth and spatio-temporally consistent repaired data matrix; S7. Verify the repaired data matrix, including spatio-temporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the parameters of the dynamic graph structure and the spatio-temporal aware graph neural network model according to the verification report.

2. The dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 1, characterized in that The specific content of S2 includes: S21. Construct an initial data graph structure , where is the initial node set, representing the data entities of the data flow in the dynamic environment, is the initial edge set, representing the association relationships between nodes, and for each node is assigned an initial feature vector and for each edge is assigned an initial weight ; S22. Calculate the change amount of node features for each node based on the dynamically collected real-time data :​ ； Among them, is the node at time is the eigenvalue, is the eigenvalue of the previous moment, is a constant to prevent the denominator from being zero; Calculate the weight change of each edge :​ ； Among them, represents two nodes connected by an edge and is the amount of change in weight, and are respectively the eigenvalues of nodes and at time and represents the modulus length of the eigenvalue, is a constant to avoid a zero denominator;​ According to the change amount of node features and the change amount of weights , dynamically adjust the initial weight of each edge ; S23. Perform threshold judgment on the adjusted edge weights and set a dynamic threshold based on the average value and standard deviation of the edge weights as follows: ； Among them, represents the number of edges in the initial edge set , is the adjusted edge weight, and is a preset adjustment coefficient. For each edge , the adjusted edge weight is compared with the dynamic threshold . When the condition is satisfied, the edge is retained in the updated edge set ; S24. According to the updated edge set , combined with the node feature change amount , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure , where and respectively represent the adjusted node set and the updated edge set; S25. Perform timeliness verification on the dynamic graph structure including whether the update frequency of nodes is synchronized with the changes in dynamic data and whether the connectivity of the edge set meets the modeling requirements; S26. During the dynamic adjustment process, record the nodes and edges that do not meet the conditions to generate an adjustment log.

3. The dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 2, characterized in that, The specific content of S3 includes: S31. Utilize the dynamic graph structure , construct a spatio-temporal aware graph neural network model, and combine the time features and the spatial adjacency matrix as inputs, where is the node set of the dynamic graph, is the edge set of the dynamic graph, and each node is assigned an initial feature vector , the time features are generated by the dynamic attribute changes of the nodes at different time steps, and the spatial adjacency matrix is defined by the edge set ; S32. Generate the spatio-temporal embedding representation of the node through the joint modeling of the node's own features, time features, and neighbor node features; S33. Aggregate the spatio-temporal embedding representations of all nodes to generate a high-dimensional spatio-temporal feature set of the dynamic graph nodes; S34. Construct a multi-layer spatio-temporal aware graph neural network, define a multi-layer recursive update rule, and calculate the recursive feature representation of nodes through a message passing mechanism. The update rule for the th layer is as follows: ； Among them, is a non-linear activation function, is the trainable weight matrix of the th layer, represents the normalized aggregation of neighbor node features, is the node in the th layer of the graph neural network, is the node in the th layer of the graph neural network; S35. Generate a dynamic graph structure through multi-layer iterative updates A high-dimensional feature matrix representing the final spatio-temporal embedding features of all nodes after multi-layer recursion .

4. A dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 3, characterized in that The specific content of S4 includes: S41. Define a feature fusion rule to splice the high-dimensional feature matrix , the initial time feature matrix and the initial spatial feature matrix according to the node index to generate an initial multi-modal feature matrix; S42. Normalize the initial multi-modal feature matrix to generate a normalized feature matrix; S43. Use an autoencoder to perform unified encoding on the normalized feature matrix to construct an autoencoder model.

5. A dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 4, characterized in that The specific content of S5 includes: S51. Input the multi-modal fusion feature matrix into a generative adversarial network, where the generative adversarial network includes a generator model and a discriminator model ; S52. Define the prediction rules of the generation model. The generation model takes the multi-modal fusion feature matrix as input and generates a preliminary repair data matrix through a deep neural network ; S53. Define a discrimination model with discrimination rules. The discrimination model takes the preliminary repair data matrix and the real data matrix as inputs, where the real data matrix is composed of data streams in a dynamic environment, and its authenticity score and are calculated through a binary classification neural network; S54. Conduct adversarial training on the generative model and the discriminative model: ； Among them, is the adversarial loss function, , are the weight coefficients, , are the bias terms, is the balance coefficient, represents the square of the Euclidean norm, represents the mean squared error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data, and respectively represent the real data and the generated data being the probability scores of being judged as real or forged, , and respectively represent the expectations on the real data, the generated data, and their joint distribution, is the adjustment coefficient, represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generation model and the discriminant model, the generation model can generate repaired data close to the real data distribution; S55. Output the preliminary repair data matrix after the generative adversarial network training .

6. A dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 5, characterized in that The specific content of S6 includes: S61. Based on the preliminary repair data matrix , construct a spatio-temporal consistency optimization model, and input the preliminary repair data matrix , the initial time feature matrix and the initial space feature matrix ; S62. Define spatio-temporal smoothness constraints, and smooth the repaired data in the preliminary repaired data matrix in the time domain and the space domain respectively: ； Among them, is the spatio-temporal smoothness loss function, is the total number of time steps, and are the balance coefficients, represents the mean square error between the repaired data and the repaired data at the previous moment, is the connection weight between nodes and in the adjacency matrix, and respectively represent the repaired data of nodes and ; S63. Define the comprehensive optimization objective function , combined with spatio-temporal consistency constraints and data repair errors: ； Among them, is the balance coefficient of the error constraint, represents the mean square error between the repaired data and the true data; S64. Use the gradient descent method to solve the comprehensive optimization objective function to obtain the repaired data matrix .

7. A dynamic data quality monitoring and repair method based on an adaptive algorithm according to claim 6, characterized in that The specific content of S7 includes: S71. Input the optimized repaired data matrix , the initial time feature matrix and the initial spatial feature matrix to construct a verification model for spatio-temporal correlation consistency check; S72. Define the time consistency verification rule and calculate the consistency index of the repaired data matrix in the time domain : ； Among them, is the norm, is the time consistency adjustment parameter, is the total number of time steps, represents the repaired data matrix at the previous moment; S73. Define the spatial consistency verification rule and calculate the consistency index of the repaired data matrix in the spatial domain : ； Among them, is the connection weight between nodes and in the adjacency matrix, is the spatial consistency adjustment parameter, represents the repair data matrix of node , represents the repair data matrix of node ; S74. Combine the time consistency and space consistency verification rules to construct a comprehensive verification index; S75. Generate a verification report according to the calculation result of the comprehensive verification index. The report content includes spatio-temporal consistency analysis and overall quality analysis of the repaired data; S76. Based on the verification report, dynamically update the topological structure of the dynamic graph structure and the parameters of the spatio-temporal aware graph neural network model.

Images