A dynamic data quality monitoring and repair method based on adaptive algorithm

Abstract

The present invention discloses a dynamic data quality monitoring and repair method based on an adaptive algorithm, comprising: S1, slicing data into segments according to a time series to generate an initial data graph structure; S2, dynamically adjusting the topological relationship of the nodes and edges of the graph using an adaptive algorithm to generate a dynamic graph structure; S3, generating a high-dimensional feature matrix by jointly modeling an initial time feature matrix and an initial space feature matrix; S4, fusing the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix to generate a multimodal fusion feature matrix; S5, inputting the multimodal fusion feature matrix into a generative adversarial network to generate a preliminary repair data matrix; S6, optimizing the preliminary repair data matrix through a differentiable loss mechanism to generate a repair data matrix; S7, verifying the repair data matrix to generate a verification report. The present invention has the advantages of high smooth consistency of data repair and good real-time verification and optimization.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a dynamic data quality monitoring and repair method based on an adaptive algorithm. Background Art

[0002] With the rapid development of big data technology and artificial intelligence, the quality monitoring and repair of dynamic data have become increasingly important in various fields. This is especially true in real-time dynamic scenarios such as the Internet of Things, financial data analysis, and intelligent transportation. The stability and accuracy of data quality directly impact system reliability and the scientific nature of decision-making. However, due to its complexity and spatiotemporal evolution, dynamic data is often susceptible to interference from missing values, outliers, and noise, posing challenges to high-quality data analysis and model application.

[0003] In existing technologies, dynamic data quality monitoring and repair typically rely on traditional static models and simple interpolation methods. These methods are inadequate when faced with complex dynamic data. Specifically, existing technologies have the following major drawbacks:

[0004] 1. Lack of dynamic graph structure modeling capabilities: Traditional methods are often based on fixed, static data structures and cannot dynamically adjust the model structure to adapt to real-time data changes. This deficiency makes traditional methods inadequate when capturing the spatiotemporal evolution of dynamic data.

[0005] 2. Insufficient support for the fusion of temporal and spatial features: In existing technologies, time series analysis and spatial feature modeling are usually performed independently. There is a lack of the ability to unify the modeling of temporal and spatial features, and it is impossible to fully reflect the multidimensional characteristics and complex associations of the data.

[0006] 3. Single data repair strategy: Existing data repair technologies often use simple interpolation or filling strategies, which fail to fully consider the global correlation and dynamic change characteristics of the data, resulting in a lack of smoothness and consistency in the repair results.

[0007] 4. Insufficient dynamic support for verification and optimization: Traditional data verification methods are mostly static verification, lacking an effective feedback mechanism for real-time dynamic changes in data, and unable to dynamically optimize data models and parameters based on verification results.

[0008] 5. Weak ability to process multimodal data: In dynamic data scenarios, data from different sources and types have multimodal characteristics, but existing technologies lack effective fusion and unified coding methods for multimodal features, and are unable to fully explore the inherent correlations of the data.

[0009] Therefore, how to provide a dynamic data quality monitoring and repair method based on an adaptive algorithm is a problem that those skilled in the art urgently need to solve. Summary of the Invention

[0010] One purpose of the present invention is to propose a dynamic data quality monitoring and repair method based on an adaptive algorithm. The present invention fully combines adaptive dynamic graph structure generation technology, spatiotemporal perception graph neural network, multimodal feature fusion analysis and generative adversarial network optimization repair strategy, and describes in detail the implementation process of dynamic data quality monitoring and repair. It has the advantages of strong dynamic modeling capabilities, accurate spatiotemporal correlation capture, high smoothness and consistency of data repair, and good real-time verification and optimization.

[0011] A dynamic data quality monitoring and repair method based on an adaptive algorithm according to an embodiment of the present invention includes the following steps:

[0012] S1. Obtain the data stream in a dynamic environment, segment the data stream according to the time series, extract the initial time feature matrix and the initial spatial feature matrix, and generate the initial data graph structure;

[0013] S2. Based on the initial data graph structure and real-time data changes, an adaptive algorithm is used to dynamically adjust the topological relationships of the graph's nodes and edges to generate a dynamic graph structure that can reflect the real-time dynamics of the data;

[0014] S3. Build a spatiotemporal-aware graph neural network model using a dynamic graph structure. By jointly modeling the initial time feature matrix and the initial spatial feature matrix, a high-dimensional feature matrix containing node temporal evolution features and spatial correlation features is generated.

[0015] S4, fusing the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix, and generating a multimodal fusion feature matrix through unified coding through multimodal feature analysis;

[0016] S5. Input the multimodal fusion feature matrix into the generative adversarial network to train the generative model, predict missing values and outliers in the data, and train the discriminative model to evaluate the authenticity of the generated data to generate a preliminary repair data matrix;

[0017] S6. Optimize the preliminary repair data matrix through a differentiable loss mechanism to generate a smooth and spatially consistent repair data matrix;

[0018] S7. Verify the repaired data matrix, including spatiotemporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the parameters of the dynamic graph structure and spatiotemporal perception graph neural network model based on the verification report.

[0019] Optionally, the S2 specifically includes:

[0020] S21. Build the initial data graph structure ,in is the initial node set, representing the data entity of the data flow in the dynamic environment, is the initial edge set, which represents the relationship between nodes, and for each node Assign initial eigenvector and each edge Assign initial weight ;

[0021] S22, calculate each node based on real-time collected dynamic data The node feature change :

[0022] ;

[0023] in, For nodes In time The eigenvalues of is the eigenvalue of the previous moment, To prevent the denominator from being zero;

[0024] Calculate each edge The weight change :

[0025] ;

[0026] in, Represents two nodes connected by an edge and The weight change of and Node and In time The eigenvalues of represents the modulus of the eigenvalue, To avoid constants with zero denominators;

[0027] According to the change of node characteristics and weight change , dynamically adjust the initial weight of each edge :

[0028] ;

[0029] in, is the initial weight of the edge, is the adjusted edge weight, and are the dynamic adjustment factors for weight change and node feature change, and Node and The node feature change amount;

[0030] S23, the adjusted edge weights Perform threshold judgment and set dynamic threshold based on the average and standard deviation of edge weights :

[0031] ;

[0032] in, Represents the initial edge set The number of edges, is the adjusted edge weight, and is the preset adjustment coefficient, for each edge , the adjusted edge weights With dynamic threshold Compare, when the conditions are met When Keep the updated edge set middle;

[0033] S24. According to the updated edge set , combined with the node feature variation , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure ,in and Represent the adjusted node set and updated edge set respectively;

[0034] S25. Dynamic graph structure Conduct timeliness verification, including whether the node update frequency is synchronized with dynamic data changes and whether the connectivity of the edge set meets the modeling requirements;

[0035] S26. During the dynamic adjustment process, nodes and edges that do not meet the conditions are recorded and an adjustment log is generated.

[0036] Optionally, the S3 specifically includes:

[0037] S31. Using dynamic graph structures , build a spatiotemporal perception graph neural network model and combine it with time features and the spatial adjacency matrix As input, is the node set of the dynamic graph, is the edge set of the dynamic graph, each node Assign initial eigenvector , time characteristics By node Dynamic attribute changes at different time steps generate spatial adjacency matrices By edge set definition;

[0038] S32, generate the spatiotemporal embedding representation of the node by jointly modeling the node's own features, time features and neighbor node features;

[0039] S33, aggregate the spatiotemporal embedding representations of all nodes to generate a high-dimensional spatiotemporal feature set of dynamic graph nodes;

[0040] S34, build a multi-layer spatiotemporal perception graph neural network, define multi-layer recursive update rules, calculate the recursive feature representation of the node through the message passing mechanism, and The update rule of the layer is:

[0041] ;

[0042] in, is a nonlinear activation function, For the The trainable weight matrix of the layer, represents the normalized aggregation of neighbor node features, For nodes In the graph neural network Feature representation in the layer, For nodes In the graph neural network Feature representation in the layer;

[0043] S35. Generate dynamic graph structure through multi-layer iterative update The high-dimensional feature matrix representing the final spatiotemporal embedding features of all nodes after multi-layer recursion .

[0044] Optionally, the S4 specifically includes:

[0045] S41, define the feature fusion rules, and transform the high-dimensional feature matrix , initial time feature matrix and the initial spatial feature matrix Splice according to the node index to generate the initial multimodal feature matrix :

[0046] ;

[0047] in, represents the feature concatenation operation of the matrix, is a multimodal feature matrix containing spatiotemporal features, time features, and spatial features;

[0048] S42, initial multimodal feature matrix Perform normalization to generate a normalized feature matrix ;

[0049] S43. Use the autoencoder to normalize the feature matrix Perform unified encoding and build an automatic encoder model. The encoding rules are as follows:

[0050] ;

[0051] in, is the multimodal fusion feature matrix after unified encoding, is the activation function of the encoder, and are the weight matrix and bias vector of the encoder respectively.

[0052] Optionally, the S5 specifically includes:

[0053] S51, multimodal fusion feature matrix Input to the Generative Adversarial Network, where the Generative Adversarial Network includes a generative model and discriminant models ;

[0054] S52. Define the generative model The prediction rules of the generative model are multimodal fusion feature matrix As input, a preliminary repair data matrix is generated through a deep neural network :

[0055] ;

[0056] in, is the activation function of the generative model, and are the weight matrix and bias vector of the generative model respectively;

[0057] S53. Define the discriminant model The discriminant rules and discriminant models are used to preliminarily repair the data matrix and the real data matrix is the input, where the real data matrix It consists of data streams in a dynamic environment, and its authenticity score is calculated through a binary classification neural network. and :

[0058] ;

[0059] in, is the activation function of the discriminant model, and are the weight matrix and bias vector of the discriminant model respectively, Represents the concatenated input of real data and generated data;

[0060] S54. Conduct adversarial training on the generative model and the discriminative model:

[0061] ;

[0062] in, To counter the loss function, 、 is the weight coefficient, 、 is the bias term, is the balance coefficient, represents the square of the Euclidean norm, Represents the mean square error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data. and Represents real data and generate data The probability score of being classified as real or fake, 、 and Represent the expectations of real data, generated data and their joint distribution, is the adjustment coefficient, Represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generative model and the discriminative model, the generative model Able to generate repair data close to the real data distribution;

[0063] S55. Output the initial repair data matrix after generating adversarial network training .

[0064] Optionally, the S6 specifically includes:

[0065] S61, based on the preliminary repair data matrix , build a spatiotemporal consistency optimization model, input the preliminary repair data matrix , initial time feature matrix and the initial spatial feature matrix ;

[0066] S62. Define spatiotemporal smoothness constraints and perform smoothing on the repair data in the preliminary repair data matrix in both the time domain and the space domain:

[0067] ;

[0068] in, is the spatiotemporal smoothness loss function, is the total number of time steps, and is the balance coefficient, Represents the mean square error between the repaired data and the repaired data at the previous moment, is the node in the adjacency matrix and The connection weight of and Represents nodes respectively and Repair data;

[0069] S63. Define comprehensive optimization objective function , combining spatiotemporal consistency constraints and data repair errors:

[0070] ;

[0071] in, is the balance coefficient of the error constraint, represents the mean square error between the repaired data and the true data;

[0072] S64, using the gradient descent method to optimize the comprehensive objective function Solve and get the repair data matrix :

[0073] ;

[0074] in, is the learning rate, represents the gradient of the loss function with respect to the repaired data matrix.

[0075] Optionally, the S7 specifically includes:

[0076] S71. Input the optimized repair data matrix , initial time feature matrix and the initial spatial feature matrix , build a verification model Used for consistency checking of spatiotemporal correlation;

[0077] S72. Define time consistency verification rules and calculate the consistency index of the repair data matrix in the time domain :

[0078] ;

[0079] in, for norm, Tuning parameters for temporal consistency, is the total number of time steps, Represents the repair data matrix of the previous moment;

[0080] S73. Define spatial consistency verification rules and calculate the consistency index of the repair data matrix in the spatial domain :

[0081] ;

[0082] in, is the node in the adjacency matrix and The connection weight of is the spatial consistency tuning parameter, Representation node The repair data matrix, Representation node The repair data matrix;

[0083] S74. Combine temporal consistency and spatial consistency verification rules to build a comprehensive verification index :

[0084] ;

[0085] in, and are the weight parameters for temporal and spatial consistency, respectively, used to dynamically balance the importance of different verification indicators;

[0086] S75, based on comprehensive verification indicators The calculation results are used to generate a verification report, which includes spatiotemporal consistency analysis and overall quality analysis of the repaired data;

[0087] S76. Based on the verification report, the dynamic graph structure The topological structure and parameters of the spatiotemporal perception graph neural network model are dynamically updated.

[0088] The beneficial effects of the present invention are:

[0089] (1) This invention provides a deep understanding of dynamic data quality and dynamic adaptability by combining adaptive dynamic graph structure generation, spatiotemporal-aware graph neural networks, multimodal feature fusion analysis, and generative adversarial network optimization and repair strategies. This enables the system to capture the spatiotemporal correlation characteristics of data and their evolution patterns, thereby effectively addressing missing values, outliers, and noise issues in dynamic data. By introducing a dynamically adjusted graph structure and efficient spatiotemporal joint modeling technology, the system can accurately extract global and local features of the data, providing comprehensive support for data quality monitoring.

[0090] (2) This paper constructs a dynamic data verification and optimization framework, utilizing spatiotemporal consistency verification indicators combined with a comprehensive optimization objective function to achieve comprehensive quality analysis and real-time optimization of repaired data. Combining the adversarial training mechanism of generative adversarial networks with the unified encoding capabilities of autoencoders, the system not only efficiently repairs data but also continuously optimizes the model structure and parameter configuration during the repair process, significantly improving the accuracy and consistency of data repair.

[0091] (3) This invention provides a multi-dimensional monitoring and real-time repair method for dynamic data quality by comprehensively integrating spatiotemporal modeling, multimodal feature analysis, and dynamic verification and optimization technologies. The system can automatically adapt to dynamic changes in data, reducing reliance on manual intervention. It can also dynamically adjust the parameter configuration of the dynamic graph structure and spatiotemporal perception model, significantly improving the intelligence and adaptability of dynamic data quality monitoring and repair, and providing a solid technical guarantee for the widespread application of dynamic data scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0093] Figure 1 This is the overall framework diagram of a dynamic data quality monitoring and repair method based on an adaptive algorithm proposed in the present invention. DETAILED DESCRIPTION

[0094] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0095] refer to Figure 1 , a dynamic data quality monitoring and repair method based on an adaptive algorithm, comprising the following steps:

[0096] S1. Obtain the data stream in a dynamic environment, segment the data stream according to the time series, extract the initial time feature matrix and the initial spatial feature matrix, and generate the initial data graph structure;

[0097] In this embodiment, S1 specifically includes:

[0098] Acquire real-time data streams from multiple data sources in a dynamic environment, preprocess the data streams, and segment and slice the preprocessed data streams in chronological order. Each slice corresponds to a fixed time interval, and the size of the time interval is set according to the requirements of the application scenario. The data stream is divided into multiple continuous time period sets, and the initial time feature matrix and initial spatial feature matrix are extracted from the time slice data of the time period set. Based on the extracted initial time feature matrix and initial spatial feature matrix, the initial data graph structure is constructed.

[0099] S2. Based on the initial data graph structure and real-time data changes, an adaptive algorithm is used to dynamically adjust the topological relationships of the graph's nodes and edges to generate a dynamic graph structure that can reflect the real-time dynamics of the data;

[0100] In this embodiment, S2 specifically includes:

[0101] S21. Build the initial data graph structure ,in is the initial node set, representing the data entity of the data flow in the dynamic environment, is the initial edge set, which represents the relationship between nodes, and for each node Assign initial eigenvector and each edge Assign initial weight ;

[0102] S22, calculate each node based on real-time collected dynamic data The node feature change :

[0103] ;

[0104] in, For nodes In time The eigenvalues of is the eigenvalue of the previous moment, To prevent the denominator from being zero;

[0105] Calculate each edge The weight change :

[0106] ;

[0107] in, Represents two nodes connected by an edge and The weight change of and Node and In time The eigenvalues of represents the modulus of the eigenvalue, To avoid constants with zero denominators;

[0108] According to the change of node characteristics and weight change , dynamically adjust the initial weight of each edge :

[0109] ;

[0110] in, is the initial weight of the edge, is the adjusted edge weight, and are the dynamic adjustment factors for weight change and node feature change, and Node and The node feature change amount;

[0111] S23, the adjusted edge weights Perform threshold judgment and set dynamic threshold based on the average and standard deviation of edge weights :

[0112] ;

[0113] in, Represents the initial edge set The number of edges, is the adjusted edge weight, and is the preset adjustment coefficient, for each edge , the adjusted edge weights With dynamic threshold Compare, when the conditions are met When Keep the updated edge set middle;

[0114] S24. According to the updated edge set , combined with the node feature variation , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure ,in and Represent the adjusted node set and updated edge set respectively;

[0115] S25. Dynamic graph structure Conduct timeliness verification, including whether the node update frequency is synchronized with dynamic data changes and whether the connectivity of the edge set meets the modeling requirements;

[0116] S26. During the dynamic adjustment process, nodes and edges that do not meet the conditions are recorded and an adjustment log is generated.

[0117] This implementation method dynamically adjusts the topological relationships of the nodes and edges of the graph through an adaptive algorithm to generate a dynamic graph structure that can reflect the real-time dynamics of the data, thereby achieving accurate capture and real-time updates of dynamic data characteristics. Combined with the dynamic calculation rules for node feature changes and edge weight changes, the sensitivity and adaptability of the graph structure to data changes are guaranteed, thereby fully reflecting the spatiotemporal correlation relationship between data. In addition, through the intelligent adjustment of dynamic thresholds and the update of edge sets, the connectivity and topological consistency of the graph are further optimized, providing a high-quality input foundation for subsequent spatiotemporal perception modeling, improving the flexibility and efficiency of dynamic graph modeling, and providing an efficient, real-time structured solution for the analysis and processing of complex dynamic data.

[0118] S3. Build a spatiotemporal-aware graph neural network model using a dynamic graph structure. By jointly modeling the initial time feature matrix and the initial spatial feature matrix, a high-dimensional feature matrix containing node temporal evolution features and spatial correlation features is generated.

[0119] In this embodiment, S3 specifically includes:

[0120] S31. Using dynamic graph structures , build a spatiotemporal perception graph neural network model and combine it with time features and the spatial adjacency matrix As input, is the node set of the dynamic graph, is the edge set of the dynamic graph, each node Assign initial eigenvector , time characteristics By node Dynamic attribute changes at different time steps generate spatial adjacency matrices By edge set definition;

[0121] S32, generate the spatiotemporal embedding representation of the node by jointly modeling the node's own features, time features and neighbor node features;

[0122] S33, aggregate the spatiotemporal embedding representations of all nodes to generate a high-dimensional spatiotemporal feature set of dynamic graph nodes;

[0123] S34, build a multi-layer spatiotemporal perception graph neural network, define multi-layer recursive update rules, calculate the recursive feature representation of the node through the message passing mechanism, and The update rule of the layer is:

[0124] ;

[0125] in, is a nonlinear activation function, For the The trainable weight matrix of the layer, represents the normalized aggregation of neighbor node features, For nodes In the graph neural network Feature representation in the layer, For nodes In the graph neural network Feature representation in the layer;

[0126] S35. Generate dynamic graph structure through multi-layer iterative update The high-dimensional feature matrix representing the final spatiotemporal embedding features of all nodes after multi-layer recursion .

[0127] This implementation utilizes a dynamic graph structure to construct a spatiotemporal perception graph neural network model. By jointly modeling initial time features and initial spatial features, it achieves in-depth capture of the temporal evolution features and spatial correlation features of dynamic data. This model not only represents node features through spatiotemporal embedding, but also effectively extracts global spatiotemporal dependencies through multi-layer recursive update rules, thereby generating a high-dimensional feature matrix that comprehensively depicts the complex data relationships and dynamic evolution laws in the dynamic graph. In this way, the model's adaptability in complex data environments is enhanced, providing accurate and efficient technical support for dynamic data quality monitoring and analysis.

[0128] S4, fusing the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix, and generating a multimodal fusion feature matrix through unified coding through multimodal feature analysis;

[0129] In this embodiment, S4 specifically includes:

[0130] S41, define the feature fusion rules, and transform the high-dimensional feature matrix , initial time feature matrix and the initial spatial feature matrix Splice according to the node index to generate the initial multimodal feature matrix :

[0131] ;

[0132] in, represents the feature concatenation operation of the matrix, is a multimodal feature matrix containing spatiotemporal features, time features, and spatial features;

[0133] S42, initial multimodal feature matrix Perform normalization to generate a normalized feature matrix ;

[0134] S43. Use the autoencoder to normalize the feature matrix Perform unified encoding and build an automatic encoder model. The encoding rules are as follows:

[0135] ;

[0136] in, is the multimodal fusion feature matrix after unified encoding, is the activation function of the encoder, and are the weight matrix and bias vector of the encoder respectively.

[0137] This implementation method generates a unified multimodal feature matrix by fusing the high-dimensional feature matrix, initial time feature matrix, and initial spatial feature matrix of the dynamic graph, and further uses an autoencoder to uniformly encode it, thereby achieving efficient integration and feature extraction of multimodal data. This method can not only capture the spatiotemporal correlation characteristics of the data, but also significantly reduce the redundant interference between feature dimensions through feature normalization and nonlinear mapping of the autoencoder, thereby enhancing the multimodal expression ability of the data, and laying the foundation for the subsequent repair and verification of the generative adversarial network. This feature fusion method improves the accuracy and robustness of dynamic data processing, and provides an efficient and flexible solution for dynamic data quality monitoring and repair.

[0138] S5. Input the multimodal fusion feature matrix into the generative adversarial network to train the generative model, predict missing values and outliers in the data, and train the discriminative model to evaluate the authenticity of the generated data to generate a preliminary repair data matrix;

[0139] In this embodiment, S5 specifically includes:

[0140] S51, multimodal fusion feature matrix Input to the Generative Adversarial Network, where the Generative Adversarial Network includes a generative model and discriminant models ;

[0141] S52. Define the generative model The prediction rules of the generative model are multimodal fusion feature matrix As input, a preliminary repair data matrix is generated through a deep neural network :

[0142] ;

[0143] in, is the activation function of the generative model, and are the weight matrix and bias vector of the generative model respectively;

[0144] S53. Define the discriminant model The discriminant rules and discriminant models are used to preliminarily repair the data matrix and the real data matrix is the input, where the real data matrix It consists of data streams in a dynamic environment, and its authenticity score is calculated through a binary classification neural network. and :

[0145] ;

[0146] in, is the activation function of the discriminant model, and are the weight matrix and bias vector of the discriminant model respectively, Represents the concatenated input of real data and generated data;

[0147] S54. Conduct adversarial training on the generative model and the discriminative model:

[0148] ;

[0149] in, To counter the loss function, 、 is the weight coefficient, 、 is the bias term, is the balance coefficient, represents the square of the Euclidean norm, Represents the mean square error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data. and Represents real data and generate data The probability score of being classified as real or fake, 、 and Represent the expectations of real data, generated data and their joint distribution, is the adjustment coefficient, Represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generative model and the discriminative model, the generative model Able to generate repair data close to the real data distribution;

[0150] S55. Output the initial repair data matrix after generating adversarial network training .

[0151] This implementation uses a generative adversarial network (GAN) to input a multimodal fusion feature matrix into a generative model and a discriminative model. The generative model predicts and repairs missing values and outliers in the data based on a deep neural network. The discriminative model evaluates the authenticity of the generated data and optimizes the generated results. Adversarial training further approximates the true data distribution. A complex adversarial loss function is used to optimize the error of the repaired data and constrain the similarity of adjacent nodes, effectively ensuring the spatiotemporal consistency and smoothness of the repaired data. This method combines the powerful predictive capabilities of the generative adversarial network with the comprehensiveness of multimodal feature fusion, improving the accuracy and stability of dynamic data repair and providing a precise and efficient solution for dynamic data quality monitoring and repair.

[0152] S6. Optimize the preliminary repair data matrix through a differentiable loss mechanism to generate a smooth and spatially consistent repair data matrix;

[0153] In this embodiment, S6 specifically includes:

[0154] S61, based on the preliminary repair data matrix , build a spatiotemporal consistency optimization model, input the preliminary repair data matrix , initial time feature matrix and the initial spatial feature matrix ;

[0155] S62. Define spatiotemporal smoothness constraints and perform smoothing on the repair data in the preliminary repair data matrix in both the time domain and the space domain:

[0156] ;

[0157] in, is the spatiotemporal smoothness loss function, is the total number of time steps, and is the balance coefficient, It represents the mean square error between the repaired data and the repaired data at the previous moment, is the node in the adjacency matrix and The connection weight of and Represents nodes respectively and Repair data;

[0158] S63. Define comprehensive optimization objective function , combining spatiotemporal consistency constraints and data repair errors:

[0159] ;

[0160] in, is the balance coefficient of the error constraint, represents the mean square error between the repaired data and the true data;

[0161] S64, using the gradient descent method to optimize the comprehensive objective function Solve and get the repair data matrix :

[0162] ;

[0163] in, is the learning rate, represents the gradient of the loss function with respect to the repaired data matrix.

[0164] This implementation constructs a spatiotemporal consistency optimization model, jointly constraining the temporal smoothness and spatial correlation of the repair data matrix. Using a comprehensive optimization objective function, iteratively optimizes the repaired data. This significantly improves the smoothness of the repaired data while effectively preserving the data's structural consistency across the spatiotemporal dimensions. By combining spatiotemporal consistency constraints with repair error optimization, the accuracy and logical consistency of the repaired data are enhanced, ensuring that the optimized data more realistically reflects the dynamic evolution of the actual data, providing strong technical support for high-quality dynamic data repair.

[0165] S7. Verify the repaired data matrix, including spatiotemporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the parameters of the dynamic graph structure and spatiotemporal perception graph neural network model based on the verification report.

[0166] In this embodiment, S7 specifically includes:

[0167] S71. Input the optimized repair data matrix , initial time feature matrix and the initial spatial feature matrix , build a verification model Used for consistency checking of spatiotemporal correlation;

[0168] S72. Define time consistency verification rules and calculate the consistency index of the repair data matrix in the time domain :

[0169] ;

[0170] in, for norm, Tuning parameters for temporal consistency, is the total number of time steps, Represents the repair data matrix of the previous moment;

[0171] S73. Define spatial consistency verification rules and calculate the consistency index of the repair data matrix in the spatial domain :

[0172] ;

[0173] in, is the node in the adjacency matrix and The connection weight of is the spatial consistency tuning parameter, Representation node The repair data matrix, Representation node The repair data matrix;

[0174] S74. Combine temporal consistency and spatial consistency verification rules to build a comprehensive verification index :

[0175] ;

[0176] in, and are the weight parameters for temporal and spatial consistency, respectively, used to dynamically balance the importance of different verification indicators;

[0177] S75, based on comprehensive verification indicators The calculation results are used to generate a verification report, which includes spatiotemporal consistency analysis and overall quality analysis of the repaired data;

[0178] S76. Based on the verification report, the dynamic graph structure The topological structure and parameters of the spatiotemporal perception graph neural network model are dynamically updated.

[0179] This implementation method constructs a spatiotemporal correlation consistency verification model to verify the consistency of the optimized repair data matrix in the time domain and the space domain. It uses the time consistency and space consistency verification rules, combined with comprehensive verification indicators, to conduct a comprehensive analysis of the spatiotemporal characteristics of the repair data. Through time consistency verification, the smoothness and continuity of the data in time evolution can be captured; through spatial consistency verification, the coordination of the data in spatial adjacency can be identified. Finally, the weights of time and space features are dynamically balanced through comprehensive verification indicators, a detailed verification report is generated, and the dynamic graph structure and model parameters are dynamically updated. This verification mechanism improves the overall consistency analysis capability of the repair data, while realizing real-time optimization of the dynamic graph and spatiotemporal perception graph neural network model, providing a reliable verification and adjustment tool for the monitoring and repair of complex dynamic data quality.

[0180] Example 1:

[0181] To verify the feasibility of this invention, we applied it to the dynamic data quality monitoring and repair scenario within an intelligent transportation system. This system monitors urban traffic data, including real-time road conditions, traffic flow statistics, vehicle speeds, and congestion conditions. This data is fundamental to urban traffic management and decision-making. However, due to the diverse data sources and complex environments, missing values, outliers, and noise are common, impacting the accuracy and efficiency of traffic management.

[0182] Recently, system administrators at a city's intelligent traffic management center discovered several issues with traffic data, including persistent data loss in certain areas, inaccurate traffic statistics, and frequent abnormal peaks. To improve data quality and support more accurate traffic optimization decisions, the management center deployed the dynamic data quality monitoring and repair method based on an adaptive algorithm, as proposed in this paper.

[0183] The intelligent transportation system first collects dynamic data from road cameras, connected vehicle sensors, and traffic management platforms. Using the adaptive algorithm of the present invention, the system constructs a dynamic graph structure that reflects the changing trends and correlations of the data in real time. Using a spatiotemporal-aware graph neural network A, the system jointly models the temporal and spatial characteristics of the traffic data, generating a high-dimensional feature matrix that incorporates global correlations.

[0184] During peak traffic periods, such as the morning rush hour on a weekday, the system performs a fusion analysis of multimodal features at key traffic nodes. Using a generative adversarial network, the system effectively corrects missing data and outliers. For example, in traffic monitoring data between Dongzhimen and Sanlitun, flow statistics were missing or abnormal during certain time periods. The system uses a repair algorithm to restore missing values to a reasonable range and optimize abnormal peaks.

[0185] The system verifies the repaired data by constructing a spatiotemporal consistency verification model. In the temporal consistency verification, the smoothness of the repaired data is analyzed to ensure that the data changes are consistent with the real trend of traffic flow; in the spatial consistency verification, the flow coordination of adjacent traffic nodes is evaluated and potential abnormal areas are discovered. Based on the verification results, the system dynamically optimizes the dynamic graph structure and model parameters, further improving the accuracy and consistency of data repair. During the one-month test, the present invention significantly improved the quality of intelligent traffic data and the responsiveness of the system. The specific test data is shown in the following table:

[0186] Table 1 Intelligent traffic data quality optimization test report

[0187]

[0188] The data table shows that the system before optimization performed poorly in key metrics such as data missing rate, outlier detection rate, and repair accuracy. In particular, the data missing rate reached 8.5%, severely impacting the Traffic Management Center's ability to accurately assess traffic flow on key sections. Furthermore, the response time was long, failing to meet the requirements of real-time optimization. The optimized system, through dynamic modeling and real-time repair strategies, significantly reduced the data missing rate to 0.4% and increased the outlier detection rate to 97.8%, effectively addressing the shortcomings in data collection.

[0189] After optimization, the system significantly improved the overall quality and coordination of data repair through a spatiotemporal consistency verification model. The data consistency verification score increased from 67.5 before optimization to 95.3. This demonstrates that the repaired data is more logical in time and space, and conforms to the dynamic evolution of actual traffic flow. Furthermore, the accuracy of data repair increased from 83.4% to 98.7%, further validating the significant effectiveness of this invention in dynamic data repair.

[0190] In actual testing, the present invention also significantly shortened data processing response time. The average repair and verification time before optimization was 45 seconds, but after optimization, it only took 8 seconds. This rapid response capability enables the system to process large amounts of dynamic data in real time, while also improving the accuracy of traffic forecasts and providing reliable support for optimized urban traffic scheduling. These data fully demonstrate the advantages of the present invention in dynamic data quality monitoring and repair.

[0191] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A dynamic data quality monitoring and repair method based on an adaptive algorithm, characterized in that: The steps include: S1. Obtain the data stream in a dynamic environment, segment the data stream according to the time series, extract the initial time feature matrix and the initial spatial feature matrix, and generate the initial data graph structure; S2. Based on the initial data graph structure and real-time data changes, an adaptive algorithm is used to dynamically adjust the topological relationships of the graph's nodes and edges to generate a dynamic graph structure that can reflect the real-time dynamics of the data; S3. Build a spatiotemporal-aware graph neural network model using a dynamic graph structure. By jointly modeling the initial time feature matrix and the initial spatial feature matrix, a high-dimensional feature matrix containing node temporal evolution features and spatial correlation features is generated. S4, fusing the high-dimensional feature matrix with the initial time feature matrix and the initial space feature matrix, and generating a multimodal fusion feature matrix through unified coding through multimodal feature analysis; S5. Input the multimodal fusion feature matrix into the generative adversarial network to train the generative model, predict missing values and outliers in the data, and train the discriminative model to evaluate the authenticity of the generated data to generate a preliminary repair data matrix; S6. Optimize the preliminary repair data matrix through a differentiable loss mechanism to generate a smooth and spatially consistent repair data matrix; S7. Verify the repaired data matrix, including spatiotemporal correlation consistency check and overall data quality analysis, generate a verification report, and dynamically update the parameters of the dynamic graph structure and spatiotemporal perception graph neural network model based on the verification report.

2. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 1, characterized in that: The S2 specifically includes: S21. Build the initial data graph structure ,in is the initial node set, representing the data entity of the data flow in the dynamic environment, is the initial edge set, which represents the relationship between nodes, and for each node Assign initial eigenvector and each edge Assign initial weight ; S22, calculate each node based on real-time collected dynamic data The node feature change : ； in, For nodes In time The eigenvalues of is the eigenvalue of the previous moment, To prevent the denominator from being zero; Calculate each edge The weight change : ； in, Represents two nodes connected by an edge and The weight change of and Node and In time The eigenvalues of represents the modulus of the eigenvalue, To avoid constants with zero denominators; According to the change of node characteristics and weight change , dynamically adjust the initial weight of each edge ; S23, the adjusted edge weights Perform threshold judgment and set dynamic threshold based on the average and standard deviation of edge weights : ； in, Represents the initial edge set The number of edges, is the adjusted edge weight, and is the preset adjustment coefficient, for each edge , the adjusted edge weights With dynamic threshold Compare, when the conditions are met When Keep the updated edge set middle; S24. According to the updated edge set , combined with the node feature variation , dynamically add or remove nodes to generate an adjusted node set , and finally form a dynamic graph structure ,in and Represent the adjusted node set and updated edge set respectively; S25. Dynamic graph structure Conduct timeliness verification, including whether the node update frequency is synchronized with dynamic data changes and whether the connectivity of the edge set meets the modeling requirements; S26. During the dynamic adjustment process, nodes and edges that do not meet the conditions are recorded and an adjustment log is generated.

3. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 2, characterized in that: The S3 specifically includes: S31. Using dynamic graph structures , build a spatiotemporal perception graph neural network model and combine it with time features and the spatial adjacency matrix As input, is the node set of the dynamic graph, is the edge set of the dynamic graph, each node Assign initial eigenvector , time characteristics By node Dynamic attribute changes at different time steps generate spatial adjacency matrices By edge set definition; S32, generate the spatiotemporal embedding representation of the node by jointly modeling the node's own features, time features and neighbor node features; S33, aggregate the spatiotemporal embedding representations of all nodes to generate a high-dimensional spatiotemporal feature set of dynamic graph nodes; S34, build a multi-layer spatiotemporal perception graph neural network, define multi-layer recursive update rules, calculate the recursive feature representation of the node through the message passing mechanism, and The update rule of the layer is: ； in, is a nonlinear activation function, For the The trainable weight matrix of the layer, represents the normalized aggregation of neighbor node features, For nodes In the graph neural network Feature representation in the layer, For nodes In the graph neural network Feature representation in the layer; S35. Generate dynamic graph structure through multi-layer iterative update The high-dimensional feature matrix representing the final spatiotemporal embedding features of all nodes after multi-layer recursion .

4. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 3, characterized in that: The S4 specifically includes: S41, define the feature fusion rules, and transform the high-dimensional feature matrix , initial time feature matrix and the initial spatial feature matrix Splice according to the node index to generate the initial multimodal feature matrix; S42, normalizing the initial multimodal feature matrix to generate a normalized feature matrix; S43. Use the autoencoder to uniformly encode the normalized feature matrix and build an autoencoder model.

5. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 4, characterized in that: The S5 specifically includes: S51, multimodal fusion feature matrix Input to the Generative Adversarial Network, where the Generative Adversarial Network includes a generative model and discriminant models ; S52. Define the generative model The prediction rules of the generative model are multimodal fusion feature matrix As input, a preliminary repair data matrix is generated through a deep neural network ; S53. Define the discriminant model The discriminant rules and discriminant models are used to preliminarily repair the data matrix and the real data matrix is the input, where the real data matrix It consists of data streams in a dynamic environment, and its authenticity score is calculated through a binary classification neural network. and ; S54. Conduct adversarial training on the generative model and the discriminative model: ； in, To counter the loss function, 、 is the weight coefficient, 、 is the bias term, is the balance coefficient, represents the square of the Euclidean norm, Represents the mean square error between the repaired data and the real data, which is used to quantify the difference between the repaired data and the real data. and Represents real data and generate data The probability score of being classified as real or fake, 、 and Represent the expectations of real data, generated data and their joint distribution, is the adjustment coefficient, Represents the similarity constraint of adjacent nodes; by optimizing the parameters of the generative model and the discriminative model, the generative model Able to generate repair data close to the real data distribution; S55. Output the initial repair data matrix after generating adversarial network training .

6. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 5, characterized in that: The S6 specifically includes: S61, based on the preliminary repair data matrix , build a spatiotemporal consistency optimization model, input the preliminary repair data matrix , initial time feature matrix and the initial spatial feature matrix ; S62. Define spatiotemporal smoothness constraints and perform smoothing on the repair data in the preliminary repair data matrix in both the time domain and the space domain: ； in, is the spatiotemporal smoothness loss function, is the total number of time steps, and is the balance coefficient, It represents the mean square error between the repaired data and the repaired data at the previous moment, is the node in the adjacency matrix and The connection weight of and Represents nodes respectively and Repair data; S63. Define comprehensive optimization objective function , combining spatiotemporal consistency constraints and data repair errors: ； in, is the balance coefficient of the error constraint, represents the mean square error between the repaired data and the true data; S64, using the gradient descent method to optimize the comprehensive objective function Solve and get the repair data matrix .

7. The method for dynamic data quality monitoring and repair based on an adaptive algorithm according to claim 6, characterized in that: The S7 specifically includes: S71. Input the optimized repair data matrix , initial time feature matrix and the initial spatial feature matrix , build a verification model Used for consistency checking of spatiotemporal correlation; S72. Define time consistency verification rules and calculate the consistency index of the repair data matrix in the time domain : ； in, for norm, Tuning parameters for temporal consistency, is the total number of time steps, Represents the repair data matrix of the previous moment; S73. Define spatial consistency verification rules and calculate the consistency index of the repair data matrix in the spatial domain : ； in, is the node in the adjacency matrix and The connection weight of is the spatial consistency tuning parameter, Representation node The repair data matrix, Representation node The repair data matrix; S74. Combine temporal consistency and spatial consistency verification rules to construct a comprehensive verification indicator; S75. Generate a verification report based on the calculation results of the comprehensive verification indicators. The report content includes spatiotemporal consistency analysis and overall quality analysis of the repaired data. S76. Based on the verification report, the topological structure of the dynamic graph structure and the parameters of the spatiotemporal perception graph neural network model are dynamically updated.

Images