A data quality monitoring method in cloud computing environment based on dynamic feedback mechanism

Abstract

The present invention discloses a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism, S1, constructing a cloud computing data set; S2, formatting the cloud computing data set; S3, making detailed adjustments based on a cloud computing service model; S4, modeling the data quality problems in the cloud computing data set as a graph structure; S5, initializing the modeled graph structure using a graph neural network; S6, automatically identifying potential abnormal transmission paths based on the real-time results output by the graph neural network; S7, adjusting the data quality monitoring strategy according to the feedback of the graph neural network; S8, performing multi-dimensional data quality assessment based on the correlation between nodes and the weight of edges; S9, outputting the assessed data quality results in a visual manner and generating a feedback report. The present invention significantly improves the real-time, flexibility and accuracy of data quality monitoring.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism. Background Art

[0002] In the existing technology, data quality monitoring in cloud computing environment faces multiple challenges. With the widespread application of cloud computing platforms, large-scale data storage, computing and transmission have become part of daily operations. Existing data quality monitoring methods mainly rely on static rules and preset parameters to monitor system logs, data storage status and transmission records, and usually judge the accuracy, consistency and timeliness of data by setting fixed thresholds or rules. However, the cloud computing environment is highly dynamic and complex, and traditional monitoring methods have significant limitations in dealing with real-time changes in data flow and data interaction between different cloud computing service models.

[0003] First, traditional static monitoring rules are difficult to adapt to the dynamic changes in the cloud computing environment. In large-scale data flows, parameters such as data traffic and data transmission delay often change in real time, and static rules cannot effectively capture these changes, resulting in data quality problems that cannot be discovered and handled in a timely manner. For example, when a node experiences a data transmission delay, the traditional monitoring system cannot promptly perceive the impact on other nodes connected to the node, thereby affecting the stability of the entire system.

[0004] Secondly, existing technologies lack flexibility and adaptability under different types of cloud computing service models. In the cloud computing service architecture, IaaS, PaaS, and SaaS service models are different, and there are significant differences in data mobility and processing methods. The existing monitoring system cannot be refined and adjusted according to the different characteristics of the service models, resulting in poor monitoring effects in some complex scenarios, especially in multi-tenant environments. The complexity of resource sharing and data flow further increases the difficulty of data quality monitoring.

[0005] In addition, when dealing with data flow relationships across nodes, existing technologies usually lack effective correlation modeling methods, making it difficult to predict the propagation path of data anomalies. As a result, when data anomalies occur at a certain node, the system cannot quickly identify and adjust the monitoring scope, resulting in an increased risk of data quality problems spreading. Ultimately, traditional monitoring methods have poor practicality and flexibility in cloud computing environments, and cannot effectively adapt to the complex and changeable cloud computing environment, which can easily lead to monitoring lags for data quality issues and affect the overall performance of the system and data reliability. Summary of the invention

[0006] One object of the present invention is to propose a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism, which significantly improves the real-time, flexibility and accuracy of data quality monitoring.

[0007] According to an embodiment of the present invention, a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism includes the following steps:

[0008] S1, collect system logs, application logs, data storage status and data transmission records of the cloud platform in the cloud computing environment in real time to build a cloud computing data set;

[0009] S2, formatting the cloud computing data set;

[0010] S3. Using various types of data in the cloud computing data set as the basis for analysis, define a set of data quality indicators for different types of cloud computing service models, and make detailed adjustments based on the cloud computing service models;

[0011] S4. Model the data quality problem in the cloud computing data set as a graph structure, where the nodes of the graph structure represent different cloud computing services or data sources, and the edges of the graph structure represent the data flow relationship between the nodes;

[0012] S5. Use graph neural network to initialize the modeled graph structure, capture the data quality association between cloud computing services through the training process, and establish a weight model between nodes and edges;

[0013] S6. Automatically identify potential abnormal transmission paths based on the real-time results output by the graph neural network. When a data anomaly occurs at a certain node, the system predicts the impact on other nodes connected to the node and adjusts the monitoring scope and monitoring strategy in a timely manner.

[0014] S7. Adjust the data quality monitoring strategy based on the feedback from the graph neural network;

[0015] S8. Perform multi-dimensional data quality assessment based on the correlation between nodes and the weight of edges. The data quality assessment dimensions include data source, storage medium, update frequency and data liquidity.

[0016] S9. Output the evaluated data quality results in a visual way and generate a feedback report, recording the anomalies that occurred during the data quality monitoring process and their handling process.

[0017] Optionally, the S1 step includes:

[0018] S11. Collect the system logs, application logs, data storage status and data transmission records of the cloud platform in real time from the cloud computing environment to build a cloud computing data set:

[0019] D={d1,d2,,dn};

[0020] Where D represents a set of n data records, d i Represents a single data record, including the following types of data:

[0021]

[0022] in, Represents the system log data of the i-th record, Represents the application log data of the i-th record, Indicates the data storage status of the i-th record, Represents the data transmission record of the i-th record;

[0023] S12: For the system log data Including CPU usage, memory usage, hard disk read and write rates, and application log data Including application startup time, execution status and application error information, data storage status Including storage device capacity, occupancy rate and storage block status, data transmission records Including data packet transmission time, transmission rate and error rate;

[0024] S13, indexing all collected data to form an indexed cloud computing data set:

[0025] D′={d′1,d′2,…,d′ n};

[0026] Among them, each d′ i It is the data record after indexing:

[0027]

[0028] in, Represents the indexed system log data. Represents the indexed application log data. Indicates the data storage status after indexing. Represents the indexed data transmission record;

[0029] S14, perform deduplication and formatting processing on the indexed cloud computing dataset D', remove duplicate data records and unify the data format, and finally construct a cloud computing dataset for data quality monitoring:

[0030] D″={d″1,d″2,…,d″ m};

[0031] Where m≤n, d″i To deduplicate and format processed data records, including standardized representation of system logs, application logs, data storage status, and data transmission records.

[0032] Optionally, the S3 step includes:

[0033] S31. Based on the cloud computing dataset D″, various types of data are used as the basis for analysis to define a preliminary data quality indicator set Q = {q1, q2,,q k}, where k is the number of quality indicators defined, q i is the i-th data quality indicator:

[0034]

[0035] Among them, f j Represents the weight factor of the data, w j is the priority weight of the data source, is the actual value of the jth instance of the i-th category data, and denote the mean and standard deviation of the jth instance, respectively, and D i is the set of data of the i-th category;

[0036] S32. For different types of cloud computing service models, including IaaS, PaaS, and SaaS, the preliminary data quality indicator set is refined and adjusted to generate an indicator set related to the service model:

[0037]

[0038] Among them, h M (t) represents the dynamic adjustment function of different cloud service models in the time dimension, It represents the second-order derivative of the quality index with respect to time, which is used to capture the accelerated change of data quality. α and β are the start and end time of the time interval, respectively, and M is the service model.

[0039] S33. Data quality indicators include the following:

[0040] The accuracy of the data, A, indicates the degree of deviation between the data and the true value;

[0041] The consistency of data, C, indicates the degree of difference between multiple data sources;

[0042] The timeliness of data, T, represents the delay time from data generation to data processing;

[0043] S34, based on different types of cloud computing service models M, define a preliminary data quality indicator set Q and a service model-related indicator set Q MThe data is integrated and the quality indicators are refined and adjusted based on the needs of the cloud computing environment to generate the final set of data quality indicators for monitoring:

[0044]

[0045] Among them, δ M is the adjustment coefficient of the service model, w M is the model weight, is the derivative of the quality index with respect to the model.

[0046] Optionally, the S4 step includes:

[0047] S41, based on the data quality indicator set Q′ finally used for monitoring M The data quality problem in cloud computing datasets is modeled as a graph structure G = (V, E), where the node set V represents different cloud computing services or data sources, the edge set E represents the data flow relationship between nodes, and the node v i ∈V and the corresponding data quality index q′ i associated;

[0048] S42, node v i The property A(v i ) includes the performance of the service represented by the node, the data processing capability and its data quality indicator set Q′ M The specific index value q′ in i ; Edge ij ∈E's property W(e ij ) represents the quality of data flow between nodes, including traffic, delay and its performance in related quality indicators. The weight of the edge w ij Defined as:

[0049]

[0050] Among them, q′ i and q′ j Respectively represent the node v i and node v j The corresponding final data quality index value, Δt ij is the data transmission delay between nodes, T max is the maximum allowed delay time in the system, and the weight w ij Represents the data quality flow relationship between two nodes;

[0051] S43, constructing data flow matrix F = [f ij ]:

[0052]

[0053] The data flow matrix is ​​used to indicate whether the data quality flow between nodes meets the monitoring requirements. ij Represents node v i and node v j Whether the mass flow relationship between them meets the preset threshold;

[0054] S44. Based on the properties of the nodes and edges of the graph structure, construct a feature matrix X of the graph structure = [x i ], where x i For node v i The feature vector includes service performance and data quality index value q′ i and data processing capabilities;

[0055] S45. Input the graph structure G and the feature matrix X of the graph structure into the graph neural network.

[0056] Optionally, the step S5 includes:

[0057] S51, initialize the graph structure G = (V, E) and the feature matrix X. The initialization process includes randomly initializing the node v i The eigenvector x i and edge ij The weight w ij ;

[0058] S52, input the graph structure G into the graph neural network, transmit and aggregate the node features through the graph convolution layer, and update the feature representation of each node. i The update rule is:

[0059]

[0060] Among them, x′ i For node v i The updated feature vector, Represents node v i The set of neighbor nodes, w ij For edge e ij The weight, deg(v i ) and deg(v j ) represent the nodes v i and node v j The degree of, W is the learnable weight matrix, σ is the activation function;

[0061] S53. The graph neural network is trained by the back propagation algorithm, and a specific loss function L based on the data quality in the cloud computing environment is defined. The specific loss function combines the accuracy, timeliness and consistency of the data and is expressed as:

[0062]

[0063] Among them, A i For node v i The true accuracy of As its predicted value, T i For node v i The authenticity and timeliness of As its predicted value, C i For node v i The true consistency of is its predicted value, α1, β1, and γ are weight factors for adjusting different data quality indicators;

[0064] S54, optimizing node feature x′ through training process i and edge weight w ij Capture the associations between cloud computing services and establish a weight model of nodes and edges based on the graph structure:

[0065]

[0066] Among them, w′ ij is the updated edge weight, reflecting the node v i and node v j The optimized data flow intensity between them, η1 is the learning rate, controlling the update step size, λ is the regularization coefficient, is the partial derivative of the loss function with respect to the edge weight, is the gradient in the time dimension, X′ i and X′ j Respectively represent the node v i and node v j Updated feature vector.

[0067] Optionally, the step S6 includes:

[0068] S61. Obtain each node v based on the output result of the graph neural network i Real-time data quality status x′ i and edge weight w′ ij , real-time calculation of the influence propagation relationship between nodes, if a node v k Data anomaly occurs, that is, the data quality indicator q′ k If it exceeds the preset threshold, the system predicts that other nodes connected to it v j Impacted by:

[0069]

[0070] Among them, I j Represents node v j Possible impact value, w′ kj For node vk and node v j The edge weight between k and x′ j The nodes v k and node v j The characteristic vector of k ) and deg(v j ) are nodes v k and node v j The degree of

[0071] S62, the system sorts all nodes that may be affected, and sorts them according to the impact value I j Arrange and determine the monitoring priority from high to low. j I j When the preset impact threshold is exceeded, the system automatically expands the monitoring range and j Add its neighboring nodes to the key monitoring list and adjust the data quality monitoring strategy in real time;

[0072] S63. According to the influence propagation relationship, the system edge weight w′ ij Dynamic adjustment is performed. If the influence propagation corresponding to some edges is strong, the edge weights are updated according to the following rules:

[0073]

[0074] Among them, w″ ij is the updated edge weight, λ1 is the learning rate, which controls the adjustment step size, The impact value I j Edge weight w′ ij The partial derivatives of are used to capture the key nodes and edges in the impact propagation path.

[0075] Optionally, the step S7 includes:

[0076] S71. Based on the real-time results of the graph neural network feedback, combined with node v i The eigenvector x′ i and edge weight w″ ij Dynamically adjust data quality monitoring strategies, modify monitoring rules, parameter configuration, and resource allocation;

[0077] S72. Calculate the abnormal probability P(v i ), based on each edge e ij The weight w″ ij and the abnormal probability of the node P(v i ) Modify the monitoring rules, adjust the relevant parameter configuration, and adjust the rules to when the weight w″ of a certain edge ij exceeds the preset threshold, and the abnormal probability P(vi ) and P(v j ) are greater than a certain critical value, increase the monitoring frequency of the edge, and allocate more computing resources and storage space for data analysis and processing;

[0078] S74. Reallocate system resources according to the real-time data flow of the cloud computing environment. i If the abnormal probability of a node and its neighbor nodes is high, the system automatically allocates more resources to the node and its associated edges;

[0079] S75. The system continuously monitors the status of nodes and edges based on the feedback results of the graph neural network, and dynamically adjusts the monitoring rules, parameter configuration and resource allocation.

[0080] The beneficial effects of the present invention are:

[0081] (1) The system of the present invention can dynamically adjust monitoring rules and strategies based on real-time collected data combined with the feedback results of the graph neural network. The mechanism can capture and predict data quality problems in real time according to different types of data flow characteristics in the cloud computing environment, and adjust monitoring parameters and ranges in a timely manner. The graph neural network is used to model the associations between nodes and edges so that the system can automatically identify potential abnormal transmission paths and achieve precise control of data transmission delays, accuracy, and consistency between nodes. This effectively solves the problem that traditional static monitoring methods cannot respond to data fluctuations in real time, and greatly improves the timeliness and accuracy of data quality monitoring in the cloud computing environment.

[0082] (2) By defining and adjusting a set of data quality indicators, the present invention enables the system to flexibly adjust the monitoring strategy according to different service models in the cloud computing architecture and to perform refined processing on the data quality indicators. Based on the feedback of the graph neural network, the system can adaptively adjust the monitoring strategy and parameter configuration to adapt to the different requirements of various service models in the cloud computing environment. This overcomes the limitations of the prior art for a single cloud service model, effectively solves the monitoring challenges brought about by inconsistent data flow characteristics in multi-tenant and complex environments, and enhances the system's adaptability to diversified cloud service architectures.

[0083] (3) Through the dynamic feedback mechanism, the system of the present invention can optimize the resource allocation strategy in real time according to the abnormal probability of the node and the data flow relationship. The system dynamically adjusts the resource allocation weight by combining the abnormal prediction value of the node with the edge weight, thereby ensuring that more computing resources are allocated to key nodes and edges, and improving the monitoring ability of potential anomalies. This not only improves the resource utilization efficiency, but also greatly enhances the flexibility and efficiency of data quality monitoring in the cloud computing environment, avoiding the monitoring blind spots or resource waste caused by uneven resource allocation in traditional systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 A flow chart of a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism proposed by the present invention;

[0086] Figure 2 This is a schematic diagram of modeling cloud computing data quality problems as a graph structure in a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism proposed by the present invention. DETAILED DESCRIPTION

[0087] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, which only illustrate the basic structure of the present invention in a schematic manner, and therefore only show the components related to the present invention.

[0088] refer to Figure 1-2 , a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism, comprising the following steps:

[0089] S1, collect system logs, application logs, data storage status and data transmission records of the cloud platform in the cloud computing environment in real time to build a cloud computing data set;

[0090] S2, formatting the cloud computing data set;

[0091] S3. Using various types of data in the cloud computing data set as the basis for analysis, define a set of data quality indicators for different types of cloud computing service models, and make detailed adjustments based on the cloud computing service models;

[0092] S4. Model the data quality problem in the cloud computing data set as a graph structure, where the nodes of the graph structure represent different cloud computing services or data sources, and the edges of the graph structure represent the data flow relationship between the nodes;

[0093] S5. Use graph neural network to initialize the modeled graph structure, capture the data quality association between cloud computing services through the training process, and establish a weight model between nodes and edges;

[0094] S6. Automatically identify potential abnormal transmission paths based on the real-time results output by the graph neural network. When a data anomaly occurs at a certain node, the system predicts the impact on other nodes connected to the node and adjusts the monitoring scope and monitoring strategy in a timely manner.

[0095] S7. Adjust the data quality monitoring strategy based on the feedback from the graph neural network;

[0096] S8. Perform multi-dimensional data quality assessment based on the correlation between nodes and the weight of edges. The data quality assessment dimensions include data source, storage medium, update frequency and data liquidity.

[0097] S9. Output the evaluated data quality results in a visual way and generate a feedback report, recording the anomalies that occurred during the data quality monitoring process and their handling process.

[0098] In this implementation, step S1 includes:

[0099] S11. Collect the system logs, application logs, data storage status and data transmission records of the cloud platform in real time from the cloud computing environment to build a cloud computing data set:

[0100] D={d1,d2,,d n};

[0101] Where D represents a set of n data records, d i Represents a single data record, including the following types of data:

[0102]

[0103] in, Represents the system log data of the i-th record, Represents the application log data of the i-th record, Indicates the data storage status of the i-th record, Represents the data transmission record of the i-th record;

[0104] S12. For system log data Including CPU usage, memory usage, hard disk read and write rates, and application log data Including application startup time, execution status and application error information, data storage status Including storage device capacity, occupancy rate and storage block status, data transmission records Including data packet transmission time, transmission rate and error rate;

[0105] S13, indexing all collected data to form an indexed cloud computing data set:

[0106] D′={d′1,d′2,…,d′ n};

[0107] Among them, each d′ i It is the data record after indexing:

[0108]

[0109] in, Represents the indexed system log data. Represents the indexed application log data. Indicates the data storage status after indexing. Represents the indexed data transmission record;

[0110] S14, perform deduplication and formatting processing on the indexed cloud computing dataset D', remove duplicate data records and unify the data format, and finally construct a cloud computing dataset for data quality monitoring:

[0111] D″={d″1,d″2,…,d″ m};

[0112] Where m≤n, d″ i To deduplicate and format processed data records, including standardized representation of system logs, application logs, data storage status, and data transmission records.

[0113] In this implementation, step S3 includes:

[0114] S31. Based on the cloud computing dataset D″, various types of data are used as the basis for analysis to define a preliminary data quality indicator set Q = {q1, q2,,q k}, where k is the number of quality indicators defined, q i is the i-th data quality indicator:

[0115]

[0116] Among them, f j Represents the weight factor of the data, w j is the priority weight of the data source, is the actual value of the jth instance of the i-th category data, and denote the mean and standard deviation of the jth instance, respectively, and D i is the set of data of the i-th category;

[0117] S32. For different types of cloud computing service models, including IaaS, PaaS, and SaaS, the preliminary data quality indicator set is refined and adjusted to generate an indicator set related to the service model:

[0118]

[0119] Among them, h M (t) represents the dynamic adjustment function of different cloud service models in the time dimension, It represents the second-order derivative of the quality index with respect to time, which is used to capture the accelerated change of data quality. α and β are the start and end time of the time interval, respectively, and M is the service model.

[0120] S33. Data quality indicators include the following:

[0121] The accuracy of the data, A, indicates the degree of deviation between the data and the true value;

[0122] The consistency of data, C, indicates the degree of difference between multiple data sources;

[0123] The timeliness of data, T, represents the delay time from data generation to data processing;

[0124] S34, based on different types of cloud computing service models M, define a preliminary data quality indicator set Q and a service model-related indicator set Q M The data is integrated and the quality indicators are refined and adjusted based on the needs of the cloud computing environment to generate the final set of data quality indicators for monitoring:

[0125]

[0126] Among them, δ M is the adjustment coefficient of the service model, w M is the model weight, is the derivative of the quality index with respect to the model.

[0127] In this embodiment, step S4 includes:

[0128] S41, based on the data quality indicator set Q′ finally used for monitoring M The data quality problem in cloud computing datasets is modeled as a graph structure G = (V, E), where the node set V represents different cloud computing services or data sources, the edge set E represents the data flow relationship between nodes, and the node v i ∈V and the corresponding data quality index q′ i associated;

[0129] S42, node v i The property A(v i ) includes the performance of the service represented by the node, the data processing capability and its data quality indicator set Q′ M The specific index value q′ in i ; Edge ij ∈E's property W(e ij ) represents the quality of data flow between nodes, including traffic, delay and its performance in related quality indicators. The weight of the edge w ij Defined as:

[0130]

[0131] Among them, q′ i and q′ j Respectively represent the node v i and node v j The corresponding final data quality index value, Δt ij is the data transmission delay between nodes, T max is the maximum allowed delay time in the system, and the weight w ij Represents the data quality flow relationship between two nodes;

[0132] S43, constructing data flow matrix F = [f ij ]:

[0133]

[0134] The data flow matrix is ​​used to indicate whether the data quality flow between nodes meets the monitoring requirements. ij Represents node v i and node v j Whether the mass flow relationship between them meets the preset threshold;

[0135] S44. Based on the properties of the nodes and edges of the graph structure, construct the feature matrix X of the graph structure = [x i ], where x i For node v i The feature vector includes service performance and data quality index value q′ i and data processing capabilities;

[0136] S45. Input the graph structure G and the feature matrix X of the graph structure into the graph neural network.

[0137] In this implementation manner, step S5 includes:

[0138] S51, initialize the graph structure G = (V, E) and the feature matrix X. The initialization process includes randomly initializing the node v i The eigenvector x i and edge ij The weight w ij ;

[0139] S52, input the graph structure G into the graph neural network, transmit and aggregate the node features through the graph convolution layer, and update the feature representation of each node. i The update rule is:

[0140]

[0141] Among them, x′ i For node vi The updated feature vector, Represents node v i The set of neighbor nodes, w ij For edge e ij The weight, deg(v i ) and deg(v j ) represent the nodes v i and node v j The degree of, W is the learnable weight matrix, σ is the activation function;

[0142] S53. The graph neural network is trained by the back propagation algorithm, and a specific loss function L based on the data quality in the cloud computing environment is defined. The specific loss function combines the accuracy, timeliness and consistency of the data and is expressed as:

[0143]

[0144] Among them, A i For node v i The true accuracy of As its predicted value, T i For node v i The authenticity and timeliness of As its predicted value, C i For node v i The true consistency of is its predicted value, α1, β1, and γ are weight factors for adjusting different data quality indicators;

[0145] S54, optimizing node feature x′ through training process i and edge weight w ij Capture the associations between cloud computing services and establish a weight model of nodes and edges based on the graph structure:

[0146]

[0147] Among them, w′ ij is the updated edge weight, reflecting the node v i and node v j The optimized data flow intensity between them, η1 is the learning rate, controlling the update step size, λ is the regularization coefficient, is the partial derivative of the loss function with respect to the edge weight, is the gradient in the time dimension, X′ i and X′ j Respectively represent the node v i and node v j Updated feature vector.

[0148] In this implementation manner, step S6 includes:

[0149] S61. Obtain each node v based on the output result of the graph neural network i Real-time data quality status x′ i and edge weight w′ ij , real-time calculation of the influence propagation relationship between nodes, if a node v k Data anomaly occurs, that is, the data quality indicator q′ k If it exceeds the preset threshold, the system predicts that other nodes connected to it v j Impacted by:

[0150]

[0151] Among them, I j Represents node v j Possible impact value, w′ kj For node v k and node v j The edge weight between k and x′ j The nodes v k and node v j The characteristic vector of k ) and deg(v j ) are nodes v k and node v j The degree of

[0152] S62, the system sorts all nodes that may be affected, and sorts them according to the impact value I j Arrange and determine the monitoring priority from high to low. j I j When the preset impact threshold is exceeded, the system automatically expands the monitoring range and j Add its neighboring nodes to the key monitoring list and adjust the data quality monitoring strategy in real time;

[0153] S63. According to the influence propagation relationship, the system edge weight w′ ij Dynamic adjustment is performed. If the influence propagation corresponding to some edges is strong, the edge weights are updated according to the following rules:

[0154]

[0155] Among them, w″ ij is the updated edge weight, λ1 is the learning rate, which controls the adjustment step size, The impact value I j Edge weight w′ ij The partial derivatives of are used to capture the key nodes and edges in the impact propagation path.

[0156] In this implementation manner, step S7 includes:

[0157] S71. Based on the real-time results of the graph neural network feedback, combined with node v i The eigenvector x′ i and edge weight w″ ij Dynamically adjust data quality monitoring strategies, modify monitoring rules, parameter configuration, and resource allocation;

[0158] S72. Calculate the abnormal probability P(v i ), based on each edge e ij The weight w″ ij and the abnormal probability of the node P(v i ) Modify the monitoring rules, adjust the relevant parameter configuration, and adjust the rules to when the weight w″ of a certain edge ij exceeds the preset threshold, and the abnormal probability P(v i ) and P(v j ) are greater than a certain critical value, increase the monitoring frequency of the edge, and allocate more computing resources and storage space for data analysis and processing;

[0159] S74. Reallocate system resources according to the real-time data flow of the cloud computing environment. i If the abnormal probability of a node and its neighbor nodes is high, the system automatically allocates more resources to the node and its associated edges;

[0160] S75. The system continuously monitors the status of nodes and edges based on the feedback results of the graph neural network, and dynamically adjusts the monitoring rules, parameter configuration and resource allocation.

[0161] Embodiment 1:

[0162] In the cloud computing environment of a large e-commerce platform, the platform processes billions of user operation records, commodity transaction records and massive amounts of logistics information every day. Since the platform has multiple different cloud service models (IaaS, PaaS, SaaS), each service model has its own specific processes and requirements for data processing. Therefore, the platform's data quality monitoring becomes particularly complex and challenging. In this dynamic and complex data environment, the platform requires real-time monitoring of data accuracy, consistency, and timeliness to ensure system stability and user experience. Traditional static data monitoring methods are stretched when dealing with such large-scale and rapidly changing data, and are unable to respond and adjust monitoring strategies in a timely manner when problems occur.

[0163] In order to improve the effectiveness of data quality monitoring, the platform introduces the data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism of the present invention. This article takes the platform as the background and describes a complete training simulation process of data quality monitoring in a cloud computing environment.

[0164] In this simulated scenario, the platform is processing a large amount of user transaction data during the "Double Eleven" shopping festival. At this time, due to the surge in traffic, the server resource utilization rate of the IaaS layer surged, and the data middleware processing speed of the PaaS layer dropped significantly, resulting in some data transmission delays and user order data anomalies at the SaaS level. Specifically, the order status update of some users is delayed. The platform needs to promptly identify and handle potential data transmission delays and order status update anomalies when data quality fluctuates to ensure data consistency and accuracy.

[0165] The system collects server usage logs of the IaaS layer, data processing logs of the PaaS layer, and order processing logs of the SaaS layer in real time to build a cloud computing data set. The data set includes data flow at each level, including CPU usage, data transmission delay, and order status update time. The system pre-processes all collected data, including deduplication and formatting operations, and generates a standardized data set.

[0166] During this process, the system will evaluate each type of data based on the different data quality indicators (accuracy, consistency and timeliness) defined by the system. For the IaaS layer, the main indicators of the system are CPU utilization and memory usage to ensure that resources are allocated reasonably; at the PaaS layer, data transmission delay and packet loss rate become key evaluation indicators; at the SaaS layer, the system will focus on monitoring the timely update and consistency of order status.

[0167] The system models data quality issues as a graph structure, defines the services and data sources of the IaaS, PaaS and SaaS layers as nodes in the graph structure, and the data flow relationships between nodes as edges. The status of each node is analyzed in real time through the graph neural network model. At this time, it is found that the status update of some orders in the SaaS layer is delayed. The system automatically identifies the nodes with abnormal order status updates through the abnormal transmission path identification method, and predicts other related nodes affected by this problem.

[0168] After identifying the problem nodes, the system adjusts the monitoring strategy based on the feedback mechanism. If the resource utilization of some nodes exceeds the preset threshold, the system will allocate more computing resources to these nodes by adjusting the resource allocation strategy. Especially when there is a delay in data transmission at the PaaS layer, the system will increase the data transmission frequency and give priority to processing data streams with larger delays.

[0169] During the "Double Eleven" period, the IaaS layer server resource utilization of the e-commerce platform reached a peak between 1 and 3 a.m., and the average CPU utilization surged from 50% to 95%. At this time, the system automatically captured these changes and issued real-time warnings. At the same time, the data transmission delay of the PaaS layer increased from the normal 30ms to 120ms, and the packet loss rate reached 5%. The order status update of the SaaS layer began to lag behind, and some orders were delayed for more than 5 minutes.

[0170] The system models and trains abnormal nodes through graph neural network models, and predicts the scope of impact on other nodes connected to these nodes. The model judges through historical data that the transmission delay of the PaaS layer will further affect the order status update of the SaaS layer. The system adjusts resource allocation in real time based on these prediction results.

[0171] At the IaaS layer, the system will prioritize allocating additional virtual machine resources to server nodes with resource bottlenecks, while increasing the data processing bandwidth at the PaaS layer and reducing data transmission delays. At the SaaS layer, the system will prioritize delayed order status updates to ensure that order accuracy and timeliness return to normal levels.

[0172] In order to verify the effectiveness of the present invention, we compared the data performance of the traditional monitoring method and the method of the present invention in the same scenario:

[0173] Traditional method: During the "Double Eleven" period, due to tight system resources and high data transmission delays, the average delay in order status updates reached 10 minutes, and the data packet loss rate reached 6%. After resource reallocation, the system took 30 minutes to resume normal operation.

[0174] The method of the present invention: in the same scenario, the system captures anomalies in real time through a dynamic feedback mechanism, the data transmission delay is controlled within 80ms, the order status update delay does not exceed 3 minutes, the data packet loss rate is reduced to 1%, the system resource allocation adjustment time is 5 minutes, and the overall recovery time is only 15 minutes.

[0175] It can be seen from the data that the method of the present invention has a faster response speed and higher resource utilization efficiency when dealing with data traffic fluctuations. In addition, through the anomaly recognition and feedback mechanism of the graph neural network model, the system can more accurately locate problem nodes and adjust the monitoring strategy in time, ensuring the accuracy and consistency of data quality.

[0176] The above simulation process proves that the method of the present invention has significant advantages in data quality monitoring in a cloud computing environment. Through the dynamic feedback mechanism, the system can respond promptly when data anomalies occur and effectively solve the problem by adjusting resource allocation and monitoring strategies. This not only improves the stability of the system and data processing efficiency, but also significantly reduces the potential losses caused by data quality problems.

[0177] The system of the present invention can dynamically adjust monitoring rules and strategies based on the real-time collected data combined with the feedback results of the graph neural network. The mechanism can capture and predict data quality problems in real time according to different types of data flow characteristics in the cloud computing environment, and adjust monitoring parameters and ranges in time. The graph neural network is used to model the associations between nodes and edges so that the system can automatically identify potential abnormal transmission paths and achieve precise control of data transmission delays, accuracy and consistency between nodes. It effectively solves the problem that traditional static monitoring methods cannot respond to data fluctuations in real time, and greatly improves the timeliness and accuracy of data quality monitoring in cloud computing environments.

[0178] By defining and adjusting a set of data quality indicators, the present invention enables the system to flexibly adjust monitoring strategies according to different service models in the cloud computing architecture and to refine the data quality indicators. Based on the feedback of the graph neural network, the system can adaptively adjust the monitoring strategies and parameter configurations to adapt to the different needs of various service models in the cloud computing environment, overcoming the limitations of the prior art for a single cloud service model, effectively solving the monitoring challenges brought about by inconsistent data flow characteristics in multi-tenant and complex environments, and enhancing the system's adaptability to diversified cloud service architectures.

[0179] Through the dynamic feedback mechanism of the present invention, the system can optimize the resource allocation strategy in real time according to the abnormal probability of the node and the data flow relationship. The system dynamically adjusts the resource allocation weight by combining the abnormal prediction value of the node with the edge weight, thereby ensuring that more computing resources are allocated to key nodes and edges, and improving the monitoring ability of potential anomalies. This not only improves the utilization efficiency of resources, but also greatly enhances the flexibility and efficiency of data quality monitoring in the cloud computing environment, avoiding the monitoring blind spots or resource waste caused by uneven resource allocation in traditional systems.

[0180] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism, characterized in that: The steps include: S1, collect system logs, application logs, data storage status and data transmission records of the cloud platform in the cloud computing environment in real time to build a cloud computing data set; S2, formatting the cloud computing data set; S3. Using various types of data in the cloud computing data set as the basis for analysis, define a set of data quality indicators for different types of cloud computing service models, and make detailed adjustments based on the cloud computing service models; S4. Model the data quality problem in the cloud computing data set as a graph structure, where the nodes of the graph structure represent different cloud computing services or data sources, and the edges of the graph structure represent the data flow relationship between the nodes; S5. Use graph neural network to initialize the modeled graph structure, capture the data quality association between cloud computing services through the training process, and establish a weight model between nodes and edges; S6. Automatically identify potential abnormal transmission paths based on the real-time results output by the graph neural network. When a data anomaly occurs at a certain node, the system predicts the impact on other nodes connected to the node and adjusts the monitoring scope and monitoring strategy in a timely manner. S7. Adjust the data quality monitoring strategy based on the feedback from the graph neural network; S8. Perform multi-dimensional data quality assessment based on the correlation between nodes and the weight of edges. The data quality assessment dimensions include data source, storage medium, update frequency and data liquidity. S9. Output the evaluated data quality results in a visual way and generate a feedback report, recording the anomalies that occurred during the data quality monitoring process and their handling process; The step S5 includes: S51, initialize the graph structure G = (V, E) and the feature matrix X. The initialization process includes randomly initializing the node v i The eigenvector x i and edge ij The weight w ij ; S52, input the graph structure G into the graph neural network, transmit and aggregate the node features through the graph convolution layer, and update the feature representation of each node. i The update rule is: Among them, x′ i For node v i The updated feature vector, Represents node v i The set of neighbor nodes, w ij For edge e ij The weight, deg(v i ) and deg(v j ) represent the nodes v i and node v j The degree of, W is the learnable weight matrix, σ is the activation function; S53. The graph neural network is trained by the back propagation algorithm, and a specific loss function L based on the data quality in the cloud computing environment is defined. The specific loss function combines the accuracy, timeliness and consistency of the data and is expressed as: Among them, A i For node v i The true accuracy of As its predicted value, T i For node v i The authenticity and timeliness of As its predicted value, C i For node v i The true consistency of is its predicted value, α1, β1, and γ are weight factors for adjusting different data quality indicators; S54, optimizing node feature x′ through training process i and edge weight w ij Capture the associations between cloud computing services and establish a weight model of nodes and edges based on the graph structure: Among them, w′ ij is the updated edge weight, reflecting the node v i and node v j The optimized data flow intensity between them, η1 is the learning rate, controlling the update step size, λ is the regularization coefficient, is the partial derivative of the loss function with respect to the edge weight, is the gradient in the time dimension, x′ i and x′ j Respectively represent the node v i and node v j Updated feature vector.

2. According to claim 1, a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism is characterized in that: The S1 step includes: S11. Collect the system logs, application logs, data storage status and data transmission records of the cloud platform in real time from the cloud computing environment to build a cloud computing data set: D={d1,d2,...,d n }; Where D represents a set of n data records, d n Represents a single data record, including the following types of data: in, Indicates the system log data of the nth record. Indicates the application log data of the nth record. Indicates the data storage status of the nth record. Indicates the data transmission record of the nth record; S12: For the system log data Including CPU usage, memory usage, hard disk read and write rates, and application log data Including application startup time, execution status and application error information, data storage status Including storage device capacity, occupancy rate and storage block status, data transmission records Including data packet transmission time, transmission rate and error rate; S13, indexing all collected data to form an indexed cloud computing data set: D′={d′1,d′2,…,d′ n }; Among them, each d′ n It is the data record after indexing: in, Represents the indexed system log data. Represents the indexed application log data. Indicates the data storage status after indexing. Represents the indexed data transmission record; S14, perform deduplication and formatting processing on the indexed cloud computing dataset D', remove duplicate data records and unify the data format, and finally construct a cloud computing dataset for data quality monitoring: D″={d″1,d″2,…,d″ m }; Where m≤n, d″ m To deduplicate and format processed data records, including standardized representation of system logs, application logs, data storage status, and data transmission records.

3. According to claim 1, a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism is characterized in that: The S3 step includes: S31. Based on the cloud computing dataset D″, various types of data are used as the basis for analysis to define a preliminary data quality indicator set Q = {q1, q2, ..., q k }, where k is the number of quality indicators defined, q k is the kth data quality indicator: Among them, f j Represents the weight factor of the data, w j is the priority weight of the data source, is the actual value of the jth instance of the i-th category data, and denote the mean and standard deviation of the jth instance, respectively, and D i is the set of data of the i-th category; S32. For different types of cloud computing service models, including IaaS, PaaS, and SaaS, the preliminary data quality indicator set is refined and adjusted to generate an indicator set related to the service model: Among them, h M (t) represents the dynamic adjustment function of different cloud service models in the time dimension, It represents the second-order derivative of the quality index with respect to time, which is used to capture the accelerated change of data quality. α and β are the start and end time of the time interval, respectively, and M is the service model. S33. Data quality indicators include the following: The accuracy of the data, A, indicates the degree of deviation between the data and the true value; The consistency of data, C, indicates the degree of difference between multiple data sources; The timeliness of data, T, represents the delay time from data generation to data processing; S34, based on different types of cloud computing service models M, define the preliminary data quality indicator set Q and the service model related indicator set Q M The data is integrated and the quality indicators are refined and adjusted based on the needs of the cloud computing environment to generate the final set of data quality indicators for monitoring: Among them, δ M is the adjustment coefficient of the service model, w M is the model weight, is the derivative of the quality index with respect to the model.

4. According to claim 1, a data quality monitoring method in a cloud computing environment based on a dynamic feedback mechanism is characterized in that: The S4 step includes: S41, based on the data quality indicator set Q′ finally used for monitoring M The data quality problem in cloud computing datasets is modeled as a graph structure G = (V, E), where the node set V represents different cloud computing services or data sources, the edge set E represents the data flow relationship between nodes, and the node v i ∈V and the corresponding data quality index q′ i associated; S42, node v i The property A(v i ) includes the performance of the service represented by the node, the data processing capability and its data quality indicator set Q′ M The specific index value q′ in i ; Edge ij ∈E's property W(e ij ) represents the quality of data flow between nodes, including traffic, delay and its performance in related quality indicators. The weight of the edge w ij Defined as: Among them, q′ i and q′ j Respectively represent the node v i and node v j The corresponding final data quality index value, Δt ij is the data transmission delay between nodes, T max is the maximum allowed delay time in the system, and the weight w ij Represents the data quality flow relationship between two nodes; S43, constructing data flow matrix F = [f ij ]: The data flow matrix is ​​used to indicate whether the data quality flow between nodes meets the monitoring requirements. ij Represents node v i and node v j Whether the mass flow relationship between them meets the preset threshold; S44. Based on the properties of the nodes and edges of the graph structure, construct a feature matrix X of the graph structure = [x i ], where x i For node v i The feature vector includes service performance and data quality index value q′ i and data processing capability information; S45. Input the graph structure G and the feature matrix X of the graph structure into the graph neural network.

5. The method for monitoring data quality in a cloud computing environment based on a dynamic feedback mechanism according to claim 1, characterized in that: The step S6 includes: S61. Obtain each node v based on the output result of the graph neural network i Real-time data quality status x′ i and edge weight w′ ij , real-time calculation of the influence propagation relationship between nodes, if a node v k Data anomaly occurs, that is, the data quality indicator q′ k If it exceeds the preset threshold, the system predicts that other nodes connected to it v j Impacted by: Among them, I j Represents node v j Possible impact value, w′ kj For node v k and node v j The edge weight between k and x′ j For each node v k and v j Real-time data quality status, deg(v k ) and deg(v j ) are nodes v k and node v j The degree of S62, the system sorts all nodes that may be affected, and sorts them according to the impact value I j Arrange from high to low and determine the monitoring priority. j I j When the preset impact threshold is exceeded, the system automatically expands the monitoring range and j Add its neighboring nodes to the key monitoring list and adjust the data quality monitoring strategy in real time; S63. According to the influence propagation relationship, the system edge weight w′ ij Dynamic adjustment is performed. If the influence propagation corresponding to some edges is strong, the edge weights are updated according to the following rules: Among them, w″ ij is the updated edge weight, λ1 is the learning rate, which controls the adjustment step size, The impact value I j Edge weight w′ ij The partial derivatives of are used to capture the key nodes and edges in the impact propagation path.

6. The method for monitoring data quality in a cloud computing environment based on a dynamic feedback mechanism according to claim 1, characterized in that: The step S7 includes: S71. Based on the real-time results of the graph neural network feedback, combined with node v i The eigenvector x′ i and edge weight w″ ij Dynamically adjust data quality monitoring strategies, modify monitoring rules, parameter configuration, and resource allocation; S72. Calculate the abnormal probability P(v i ), based on each edge e ij The weight w″ ij and the abnormal probability of the node P(v i ) Modify the monitoring rules, adjust the relevant parameter configuration, and adjust the rules to when the weight w″ of a certain edge ij exceeds the preset threshold, and the abnormal probability P(v i ) and P(v j ) are greater than a certain critical value, increase the monitoring frequency of the edge and allocate more computing resources and storage space for data analysis and processing; S74. Reallocate system resources according to the real-time data flow of the cloud computing environment. i If the abnormal probability of a node and its neighbor nodes is high, the system automatically allocates more resources to the node and its associated edges; S75. The system continuously monitors the status of nodes and edges based on the feedback results of the graph neural network, and dynamically adjusts the monitoring rules, parameter configuration and resource allocation.

Images