An intelligent data quality evaluation method, device and medium based on multi-modal analysis

Abstract

This invention relates to the field of data processing technology, and discloses an intelligent data quality assessment method, device, and medium based on multimodal analysis. The method includes: constructing a propagation path for data quality issues among heterogeneous systems; identifying key related nodes in the propagation of data quality issues based on the propagation path and a pre-constructed multimodal data quality issue impact knowledge graph; mining dynamic causal relationship patterns between key related nodes and downstream application performance by combining the historical operational indicator data of the key related nodes with those of downstream applications; and generating intelligent quantitative assessment results based on the dynamic causal relationship patterns and real-time acquired downstream application performance monitoring data. This invention, by integrating multi-source heterogeneous log data with multimodal knowledge graph analysis, achieves accurate tracing of data quality anomalies among heterogeneous systems, intelligent identification of key nodes, and real-time quantitative assessment of their impact on downstream application performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and more specifically, to an intelligent data quality assessment method, device, and medium based on multimodal analysis. Background Technology

[0002] With the rapid development of information technology, the scale of data in information systems within enterprises and organizations is growing exponentially, and data sources are becoming increasingly diverse and heterogeneous, leading to a growing number of data quality issues. These issues manifest as missing data, data errors, and abnormal data fluctuations, which not only directly affect the accuracy of data analysis and decision-making, but also, as data flows and is shared between systems, can further spread, impacting the stable operation of downstream business application systems and even causing significant economic losses.

[0003] Currently, most common data quality assessment methods are static and rule-based, relying on manually defined quality indicators, data format validation, and simple statistical analysis to identify data quality issues. While these methods may be effective in single systems or specific scenarios, they struggle to effectively identify cross-system data quality propagation paths and pinpoint key nodes in the propagation of data quality problems in complex multi-system environments and real-time dynamic data interaction scenarios. Furthermore, existing data quality assessment methods often lack in-depth analysis of the causal relationship between data quality anomalies and business system performance, making it difficult to intuitively and effectively assess the specific impact of data quality issues on the actual operational performance of business systems.

[0004] Therefore, existing technologies cannot meet the needs of real-time, accurate, and intelligent data quality assessment in complex system environments. There is an urgent need to propose a more efficient, accurate, and intelligent data quality assessment method to effectively support enterprise data governance and the optimized operation of business systems. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an intelligent data quality assessment method, device, and medium based on multimodal analysis.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A smart data quality assessment method based on multimodal analysis, the method comprising:

[0008] Based on the data access logs of the cloud platform and the operation logs of downstream applications, the propagation path of data quality problems between heterogeneous systems is constructed;

[0009] Based on the propagation path and the pre-constructed multimodal data quality problem impact knowledge graph, key related nodes in the propagation of data quality problems are identified;

[0010] By combining the historical performance metrics data of the key related nodes and downstream applications, we can explore dynamic causal relationship patterns between the performance of the key related nodes and downstream applications.

[0011] Based on the dynamic causal relationship pattern and the real-time acquired downstream application performance monitoring data, an intelligent quantitative evaluation result is generated to assess the impact of data quality on the actual performance of downstream applications.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor, when executing the computer program, implements the intelligent data quality assessment method based on multimodal analysis described above.

[0013] A computer-readable storage medium storing a computer program, which, when executed, implements the intelligent data quality assessment method based on multimodal analysis described above.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0015] This invention constructs a cross-system data quality anomaly propagation path and integrates data access logs, application operation logs, and system topology to accurately locate and trace data quality problems across heterogeneous systems. This overcomes the problem that traditional single-system detection methods cannot clearly identify the path and source of data quality anomalies during cross-system propagation, and significantly improves the accuracy and efficiency of data quality anomaly location.

[0016] This invention constructs a knowledge graph of the impact of multimodal data quality problems and, based on the fusion analysis of graph topology and node semantic vector representation, accurately identifies key related nodes in the propagation process of data quality anomalies. This effectively solves the problem that existing technologies rely solely on single-dimensional indicators, making it difficult to comprehensively and accurately identify key nodes, and improves the pertinence and efficiency of data quality problem analysis and governance.

[0017] This invention combines dynamic causal relationship pattern mining with real-time performance monitoring data to quantify the impact of data quality anomalies on the actual performance of downstream applications in real time. It achieves real-time and intelligent assessment of the performance impact caused by data quality anomalies, overcoming the shortcomings of traditional methods that cannot assess the impact of data quality problems on business performance in real time and quantitatively, and significantly improving the real-time performance and decision support capabilities of enterprise data governance. Attached Figure Description

[0018] Figure 1A flowchart of an intelligent data quality assessment method based on multimodal analysis provided by the present invention;

[0019] Figure 2 A schematic diagram of the structure of an electronic device provided by the present invention;

[0020] Figure 3 This is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Example 1

[0023] Please see Figure 1 As shown in the figure, this embodiment discloses an intelligent data quality assessment method based on multimodal analysis, the method comprising:

[0024] S101: Based on the data access logs of the cloud platform and the operation logs of downstream applications, construct the propagation path of data quality problems between heterogeneous systems;

[0025] In a specific implementation, the propagation path of data quality issues between heterogeneous systems includes:

[0026] Extract log entries that record data access anomalies from the cloud platform's data access logs, and identify the data resource identifier and access source system corresponding to the anomaly event;

[0027] Specifically, in this embodiment, the data access logs of the cloud platform originate from the log collection agents set up in the distributed cloud computing platform. These agents are deployed on various data service nodes, collect the access status of the data service nodes to the external data interfaces in real time, and push them to the centralized log repository in a unified structured log format in real time.

[0028] Furthermore, the log entries that record abnormal data access specifically include, but are not limited to, the following characteristics:

[0029] Access response status codes explicitly marked in the log entries as 4XX or 5XX indicate client or server errors;

[0030] The response time for the data access request recorded in the log entry exceeds a preset threshold (e.g., more than 500 milliseconds).

[0031] The log entries contain multiple consecutive abnormal requests to the same data resource within a short period of time, with the frequency of abnormal requests exceeding the normal range (e.g., more than twice the historical average).

[0032] The specific method used to identify the data resource identifier and access source system corresponding to an abnormal event is as follows:

[0033] First, the data resource identifier is determined based on the unique resource identifier (such as a Unique Resource Locator URL or Interface ID) contained in the log item;

[0034] Secondly, the specific source system is determined by the access party's IP, user account identifier, or system identifier fields in the log entries. The source system specifically includes heterogeneous system nodes within the enterprise (such as data processing platforms, customer management systems, business analysis platforms, etc.).

[0035] Extract log entries that record data processing anomalies from the downstream application's runtime logs, and identify the data resource identifiers and target application systems corresponding to the anomaly events;

[0036] In the specific implementation process, the downstream application systems include, but are not limited to, various business application systems such as enterprise internal risk control systems, marketing analysis systems, and financial accounting systems. To achieve unified collection of operation logs, this embodiment pre-deploys a log collection module (such as open-source components Logstash, Fluentd, or enterprise-developed log collection tools) within each application system to collect and push operation logs to unified storage in real time.

[0037] It should be further explained that log entries recording data processing anomalies specifically include the following characteristics:

[0038] The data processing exception or failure code is explicitly marked in the log entry;

[0039] The running time or execution latency of the corresponding data processing task exceeds the set normal processing time (for example, the task execution time exceeds 50% of the average time of historical tasks).

[0040] Within a short period of time, processing tasks for the same data resource failed or encountered anomalies multiple times, with the number of failures significantly exceeding the historical average.

[0041] When identifying the data resource identifier and target application system corresponding to an abnormal event, the specific method is as follows:

[0042] First, the corresponding data resource is determined by the data resource identifier (such as data interface name, data file path, or data ID) in the log item;

[0043] Secondly, the target application system can be identified through the application system identification field (such as system name, module name, or service name) in the log entries.

[0044] Based on the matching relationship of data resource identifiers and the time sequence of occurrence of abnormal events recorded in the logs, the cross-system propagation relationship between abnormal data access events and abnormal data processing events is determined.

[0045] The determination of cross-system propagation relationships includes:

[0046] Based on the time range of the abnormal events in the logs, filter out data access abnormal events and data processing abnormal events that meet the time overlap condition;

[0047] In specific implementation, the time overlap window set in this embodiment is 1 minute. That is, starting from the timestamp of a certain data access anomaly event, a window interval of ±30 seconds is determined, and data processing anomaly events appearing in this interval are filtered. Only event pairs that meet this condition can proceed to subsequent steps to avoid the association of irrelevant events.

[0048] For example, if a data access exception occurs at 10:15:30, the time overlap condition is only considered met if the corresponding data processing exception occurs between 10:15:00 and 10:16:00.

[0049] By utilizing system topology information, the selected events are matched with upstream and downstream nodes according to data resource identifiers to form cross-system event dependencies;

[0050] In practice, a system topology database for enterprise data flow is established in advance. The system topology clearly records information such as the data transmission direction, dependency relationship, and data interface relationship between various heterogeneous systems within the enterprise.

[0051] Subsequently, the system topology database is used to verify the upstream and downstream logical relationships of the selected event pairs. That is, it is determined whether the access source system with abnormal data access and the target application system with abnormal data processing conform to the predefined enterprise data flow topology. If they conform, it is confirmed that a clear cross-system event dependency relationship has been formed. Otherwise, the event pair is excluded to avoid misjudgment and redundant paths.

[0052] For example, if the source system of the data access anomaly is "Enterprise Data Warehouse A", the target application system of the data processing anomaly is "Marketing Analytics Platform B", and the topology database clearly records that Data Warehouse A is indeed the upstream of Marketing Analytics Platform B, then the event pair constitutes a valid upstream and downstream event dependency relationship.

[0053] Based on cross-system event dependencies, generate cross-system propagation relationships between data access anomaly events and data processing anomaly events;

[0054] In practice, a directed graph structure is used to represent cross-system propagation relationships, where nodes are the enterprise's heterogeneous systems (such as data warehouses, analysis platforms, risk control systems, etc.), and edges represent the propagation direction of abnormal events. Furthermore, each edge is marked with the specific time difference, propagation frequency, or other data attributes that are helpful for subsequent analysis, forming a preliminary cross-system propagation relationship graph.

[0055] Based on the cross-system propagation relationship, a propagation path for data quality issues between heterogeneous systems is generated;

[0056] In the specific implementation process, in order to optimize and clarify the propagation path of data quality issues and avoid path redundancy or contradictions, the specific methods are as follows:

[0057] For the preliminary cross-system propagation relationship graph, a path weight evaluation method is applied to optimize the path and remove redundant or low-frequency paths;

[0058] Specifically, the path weight evaluation method in this embodiment comprehensively considers factors such as propagation frequency (number of occurrences) and average propagation delay for calculation. An example formula is as follows:

[0059]

[0060] In the formula: Indicates the frequency of path occurrence. This represents the maximum propagation frequency in the path set. Indicates the average propagation delay along the path. For the set reference delay, higher weights indicate more efficient paths.

[0061] For cases of conflicting path directions (such as bidirectional or circular propagation), a topological sorting algorithm is used to resolve path conflicts, ensuring the unidirectionality and acyclicity of the propagation path.

[0062] In practice, the depth-first search (DFS) topological sorting method is applied to traverse and sort the propagation relationship graph. Based on the sorting results, directional contradictions or closed loop paths are eliminated to form a propagation path graph with a unique propagation direction.

[0063] Finally, based on the optimized path weight sorting and topology sorting results, the clear propagation path of data quality problems among the heterogeneous systems of the enterprise is determined and output. The specific example path format is: "Data Warehouse A → Marketing Analysis Platform B → Risk Control System C".

[0064] This enables precise location and tracking analysis of data quality issues propagating across multiple heterogeneous systems within an enterprise, ensuring timely and effective control and handling of these issues.

[0065] S102: Based on the propagation path and the pre-constructed multimodal data quality problem impact knowledge graph, identify key related nodes in the propagation of data quality problems;

[0066] In a specific implementation, the method by which multimodal data quality issues affect the construction of knowledge graphs includes:

[0067] Data access logs, downstream application operation logs, and system topology information are obtained from the cloud platform to extract system node characteristics and data quality anomaly event characteristics.

[0068] Specifically, in this embodiment, the cloud platform periodically obtains the following multi-source heterogeneous data from all heterogeneous systems within the enterprise through a pre-deployed log collection agent and API interface:

[0069] Data access logs record external access information for each data node within the enterprise, including request time, requester system identifier, data resource identifier, response time, status code, etc. An example log structure is shown below:

[0070]

[0071] Application runtime logs record the data processing status of each downstream application system, including task ID, data resource identifier, start and end times, processing result status, etc. An example structure is shown below:

[0072]

[0073] System topology information records in detail the static connections, data flow directions, dependencies, and interaction interfaces between heterogeneous systems within the enterprise, forming a unified topology database. An example of specific topology information is shown below:

[0074]

[0075] Based on this, further extract system node features and data quality anomaly event features:

[0076] System node characteristics include: node location (relative position in the topology), historical data traffic, processing capacity indicators (such as CPU, memory utilization, etc.), response latency, etc.

[0077] Characteristics of data quality anomalies include: event frequency, average response time to anomalies, time window of anomaly occurrence, historical anomaly frequency and trend characteristics, etc.

[0078] By using a unified data preprocessing module, data features from different sources are standardized, giving them a unified data representation (such as structured numerical matrices or vectors).

[0079] Based on the temporal co-occurrence relationship of node characteristics during the occurrence of abnormal events and the interaction relationship between systems, the propagation correlation of data quality problems among nodes is determined;

[0080] In practice, the propagation relationship of data quality problems between nodes is determined by a combination of two parts:

[0081] First, based on the temporal co-occurrence relationship during the occurrence of abnormal events, that is, analyzing the time difference between overlapping or consecutive occurrences of abnormal events within their time windows, a temporal co-occurrence coefficient (TC) is defined as follows:

[0082]

[0083] In the formula: and Let i and j be the sets of time windows for abnormal events, respectively. This indicates the duration of the intersection between the two. The TC value indicates the duration of the union; a high TC value indicates a high temporal correlation between events between nodes.

[0084] Secondly, the propagation direction between nodes is determined by combining the interaction relationship between the systems, and the upstream and downstream relationship between nodes is clarified by using the system topology information. Only the correlation relationship with high temporal co-occurrence coefficient between upstream and downstream nodes is retained, and unreasonable or non-existent correlations in the topology are excluded, thereby forming the propagation correlation relationship between nodes.

[0085] For example, if the temporal co-occurrence coefficient of nodes SYS_A and SYS_B is 0.8, and the topology shows that SYS_A is indeed an upstream of SYS_B, then a propagation relationship between SYS_A and SYS_B is confirmed.

[0086] The data quality problem propagation correlation is integrated with the performance index correlation features contained in the system node features in a multimodal manner to generate a multimodal correlation structure between nodes;

[0087] The step of multimodal integration of the data quality problem propagation correlation with the performance index correlation features contained in the system node features includes:

[0088] Based on the propagation correlation of data quality issues, the propagation time sequence features between nodes are extracted, and the change response features of node performance indicators are extracted from the system node features.

[0089] In this embodiment, the inter-node propagation timing characteristics are defined as the average time delay, delay fluctuation range, and propagation delay trend characteristics (such as an upward or downward trend) of an abnormal event propagating from an upstream node to a downstream node. Specific timing characteristic examples are as follows:

[0090]

[0091] The specific response characteristics of changes in node performance metrics include: the magnitude of changes in node CPU utilization, the magnitude of changes in memory utilization, and the proportion of abnormal changes in response time, as shown in the following examples:

[0092]

[0093] The correlation strength between nodes is determined based on the matching relationship between propagation timing characteristics and the change response characteristics of node performance indicators;

[0094] In practice, the correlation strength between nodes is determined by calculating the correlation coefficient between propagation time-series characteristics and performance index change characteristics (e.g., Pearson correlation coefficient or Spearman rank correlation coefficient). A specific formula example is shown below:

[0095]

[0096] In the formula: This represents the i-th sample value of the propagation time sequence feature; This represents the i-th sample value of the corresponding node performance change response feature; and This represents the sample mean for the corresponding feature.

[0097] A higher correlation strength (r) value (e.g., r>0.7) indicates a stronger correlation between anomaly propagation and performance changes between nodes.

[0098] The multimodal association structure between nodes is generated based on the association strength between nodes;

[0099] In practice, based on the determined association strength, a weighted directed graph structure is constructed with nodes as points and association strength as weighted edges. The weight between nodes directly reflects the degree of influence of anomaly propagation, forming a multimodal association structure between nodes.

[0100] For example, the final multimodal association structure can be formally represented as a weighted adjacency matrix, where each element represents the association strength between nodes.

[0101] Based on the multimodal association structure between the nodes, a knowledge graph of the impact of multimodal data quality issues is generated;

[0102] In a specific implementation, the method by which multimodal data quality issues affect the generation of knowledge graphs includes:

[0103] Specifically, the multimodal data quality problem impact knowledge graph constructed in this embodiment is an intelligent and interpretable graph data structure based on knowledge graph technology, multi-source heterogeneous data fusion and network structure representation. It is used to clarify the propagation and impact paths of data quality problems between different system nodes, and assist in the precise management of data quality and the tracing of the root causes of problems.

[0104] The system nodes, data resources, and relationships in the multimodal association structure between nodes are defined as entity and relationship types in the graph;

[0105] In the specific implementation process, the constituent elements of the knowledge graph are first determined, including entity types and relation types:

[0106] (1) Entity types include, but are not limited to:

[0107] System nodes (such as SYS_A, SYS_B): represent the instantiation nodes of heterogeneous systems or applications;

[0108] Data resources (such as RES_001, RES_002): represent data objects that are accessed and processed by system nodes.

[0109] (2) Relationship types include, but are not limited to:

[0110] Anomaly propagation relationship: This indicates the propagation of data quality anomalies from upstream nodes to downstream nodes;

[0111] Performance correlation: This refers to the correlation between nodes in which performance changes affect each other.

[0112] Examples of the above entity and relation types are shown in the table below:

[0113]

[0114] Based on the association strength and propagation direction in the multimodal association structure between nodes, the basic triplet structure of the knowledge graph is constructed.

[0115] In practice, the basic data is constructed using standardized knowledge graph triples (Head-Relation-Tail, or HRT for short):

[0116] For example, the relational triplet structure for propagating data quality anomalies from node SYS_A to SYS_B is represented as:

[0117] Anomaly propagation triple: (SYS_A, anomaly propagation, SYS_B, association strength = 0.85);

[0118] The performance-related triplet structure is represented as follows:

[0119] Performance-related triples: (SYS_A, performance-related, SYS_B, correlation strength = 0.78);

[0120] Furthermore, these triples can be stored in a graph database (such as Neo4j) as nodes and relationships, with the association strength as the weight of the edges, enabling the knowledge graph to have quantitative analysis capabilities.

[0121] By integrating multimodal data features into graph entities and relation attributes, a knowledge graph with multimodal semantic features is formed.

[0122] In specific implementation, this embodiment further adds the node features and abnormal event features extracted in the previous steps to the corresponding graph entities or relations in the form of attributes, enriching the semantic information of the graph. A specific example is as follows:

[0123] Additional attributes of system node entities (such as SYS_A) include: CPU utilization change rate (e.g., +20%), response latency change (e.g., an average increase of 100ms), and memory utilization change rate (e.g., +10%).

[0124] Additional attributes for data resource entities (such as RES_001) include: access frequency (e.g., 1200 times per hour), abnormal access rate (e.g., 3.5%), and resource importance level (e.g., high, medium, low).

[0125] Additional attributes of relationships (such as anomalous propagation relationships) include: mean propagation delay (e.g., 20 seconds), variance of propagation delay (e.g., 5 seconds), and propagation trend (e.g., upward trend).

[0126] Multi-dimensional semantic embedding is performed on the constructed multimodal knowledge graph to generate semantic vector representations of nodes and relationships;

[0127] Furthermore, to improve the efficiency and accuracy of subsequent analysis, this embodiment employs graph representation learning techniques, such as GraphSAGE, TransE, or GAT (Graph Attention Network) algorithms, to vectorize the multimodal knowledge graph. Specifically, in implementation:

[0128] First, using a graph embedding algorithm, the vector representation of nodes is learned by taking the multimodal attribute features of node entities as input:

[0129] The following is an example using the GraphSAGE method, where the node representation update formula is:

[0130]

[0131] In the formula: Here is the vector representation of node v at the k-th layer; the AGGREGATE function is a node aggregation function, such as mean aggregation or pooling aggregation. Let be the learnable weight matrix of the k-th layer; It is a non-linear activation function, such as the ReLU function.

[0132] Secondly, the relationship is vectorized using methods such as the TransE algorithm: the formula for the TransE relationship vector representation method is as follows: In the formula: h is the head entity vector; r is the relation vector; t is the tail entity vector.

[0133] Through the semantic embedding steps described above, the nodes and relationships in the knowledge graph all have clear low-dimensional semantic vector representations, which can effectively support subsequent key node identification, anomaly propagation path tracing, and intelligent reasoning analysis of data quality issues.

[0134] Furthermore, after constructing the knowledge graph of the impact of multimodal data quality issues, this embodiment identifies key related nodes in the propagation of data quality issues by comprehensively analyzing the topological and semantic features of nodes, as follows:

[0135] First, based on the cross-system anomaly propagation direction determined by the propagation path, the corresponding propagation path is located in the knowledge graph of the impact of multimodal data quality problems, and all system nodes on the path are extracted to form a candidate node set.

[0136] Secondly, for each node in the candidate node set, its structural importance index is calculated, including the node's degree centrality, betweenness centrality, and compact centrality. Among them, degree centrality reflects the degree of direct connection between the node and other nodes; betweenness centrality reflects the node's key role as a bridge in the propagation path; and compact centrality reflects the degree of proximity of the node to other nodes in the propagation structure.

[0137] Simultaneously, using pre-generated semantic vector representations of nodes in the graph, the average semantic similarity between each node and its directly connected neighbors is calculated to characterize the degree of semantic aggregation of nodes. The specific calculation method is cosine similarity:

[0138]

[0139] In the formula: Let be the semantic vector representation of node 𝑢 and its neighbor node 𝑣. This represents the semantic similarity between nodes.

[0140] Subsequently, by weighting the structural importance and semantic aggregation degree of the fused nodes, the comprehensive importance score of each node is calculated. The specific calculation formula is as follows:

[0141]

[0142] In the formula: The weighting factors were determined experimentally. , , are the normalized values ​​of the degree centrality, betweenness centrality, and compact centrality of node 𝑢, respectively. This is the normalized value of the average semantic similarity of node 𝑢.

[0143] Finally, all nodes in the candidate node set are sorted from high to low according to their comprehensive importance scores. Several nodes with scores higher than the preset threshold or those with high rankings are selected as key related nodes in the propagation of data quality problems, thereby providing accurate input for subsequent dynamic causal relationship pattern mining.

[0144] S103: Combining the historical operating index data of the key related nodes and downstream applications, we can explore the dynamic causal relationship pattern between the performance of the key related nodes and downstream applications.

[0145] In a specific implementation, the dynamic causal relationship pattern between key related nodes and downstream application performance includes:

[0146] Obtain historical records of data quality anomaly events corresponding to key related nodes, as well as historical operational performance metrics data for downstream applications;

[0147] Specifically, in this embodiment, the key associated nodes are the system nodes identified in step S102 that play a significant role in the propagation of data quality anomalies. The historical data quality anomaly event records include, but are not limited to, detailed indicators such as the frequency of historical data access anomalies, the duration of anomalies, the delay in anomaly response, and changes in the size of anomaly data.

[0148] In practice, data is first collected from the following two sources:

[0149] (1) Historical records of data quality anomalies in key related nodes, for example:

[0150] The frequency of abnormal data access events per day over the past 30 consecutive days for node "SYS_X";

[0151] The duration of the abnormal state of node "SYS_X" (e.g., each abnormal state lasts 10 to 60 seconds).

[0152] Average abnormal response latency (e.g., the average response latency over the past month is 120ms);

[0153] Data size changes during each abnormal event (e.g., a sudden 20% increase in data size);

[0154] (2) Historical operating performance data of downstream applications, including:

[0155] Application service response latency (e.g., response latency increases to 200ms);

[0156] Application service processing data throughput (e.g., throughput decreases by 10%).

[0157] Application service abnormal call frequency (e.g., daily abnormal call rate of 3%).

[0158] The specific method for obtaining data is to export historical running data through log analysis and performance monitoring platforms (such as Prometheus and Grafana), and then perform data preprocessing (such as missing value imputation and outlier removal) to generate standardized historical data sequences for subsequent causal relationship pattern mining.

[0159] Based on the temporal relationship between data quality anomaly events and performance index data, initial causal association candidate patterns are determined;

[0160] In specific implementation, the method for determining the initial causal association candidate pattern proposed in this embodiment is fully disclosed as follows:

[0161] For historical records of data quality anomalies and performance index data, time series alignment is performed based on timestamps to ensure the temporal synchronization between the two data series.

[0162] For example, aligning two data sources using a fixed sliding window (e.g., 10 minutes) as the time unit:

[0163] Example of a data quality anomaly event sequence (SYS_X node): [T, T+10min, T+20min, ..];

[0164] Example of a performance metric data sequence (application service Y): [T, T+10min, T+20min, ...].

[0165] The Granger causality test method is used to calculate the causal relationship test statistic between the data quality anomaly event sequence and the performance index data sequence, and to determine the statistically significant initial causal association candidate patterns.

[0166] Specifically, the formula for determining candidate patterns using the Granger causality test is as follows:

[0167]

[0168] In the formula: The value of the performance index data at time t; The value of the data quality anomaly event at time tj; , The coefficients of the model to be estimated are... is the random error term; p and q are the lag orders, and the optimal orders are determined by the Bayesian Information Criterion (BIC).

[0169] By conducting significance tests (such as the F-test), with a significance level of α=0.05, a set of initial causal association candidate patterns that are statistically significant is selected.

[0170] The effectiveness of the initial causal association candidate patterns is evaluated using the causal pattern stability analysis method, and stable causal association patterns that meet the temporal consistency constraint are selected.

[0171] The effectiveness evaluation of the initial causal association candidate patterns includes:

[0172] Based on the temporal characteristics of the initial causal relationship candidate patterns, we verify whether the temporal order constraints of events and performance indicators are met, and remove candidate patterns that do not meet the temporal constraints.

[0173] In specific implementation, the "timing constraints" are defined as follows:

[0174] The timing constraint requires that the occurrence of data quality anomalies must strictly precede changes in performance metrics;

[0175] By calculating the time lag difference (ΔT), ΔT must satisfy the following condition:

[0176]

[0177] in, A reasonable lag time range determined by experience, for example =1min, =30min;

[0178] If a candidate pattern does not meet the above timing constraints, it is removed from the candidate pattern set.

[0179] By using independent historical data fragments, the recurrence of candidate patterns that meet the time series constraints is verified, and candidate patterns with recurrence stability are selected.

[0180] In specific implementation, the verification method repeatedly used in this embodiment is as follows:

[0181] Divide historical data into multiple independent data segments (e.g., divide the data from the past month into 10 independent segments, each segment lasting 3 days).

[0182] Validate whether the candidate pattern appears significantly in each individual segment;

[0183] If a candidate pattern appears significantly and repeatedly in more than 70% (e.g., at least 7 out of 10) of the data segments, then the candidate pattern is defined as having "repetition stability" and is retained in the candidate pattern set.

[0184] Conversely, if a candidate pattern does not meet the above repetition criteria, it is removed from the set.

[0185] Based on candidate patterns with repeatability stability, generate stable causal association patterns that satisfy temporal consistency constraints;

[0186] Furthermore, the candidate patterns verified through the aforementioned recurrence will be organized and stored in a "pattern-stability score" format. The stability score for each pattern will be calculated as follows:

[0187]

[0188] The scores above reflect the stability of the patterns in historical data segments, ultimately forming a set of stable causal association patterns, as shown in the following example:

[0189]

[0190] Based on a stable causal relationship pattern, a dynamic causal relationship pattern between key related nodes and downstream application performance is generated.

[0191] In practical implementation, this embodiment proposes to further extract dynamic trend characteristics (such as the frequency trend, stability trend, and time lag trend) between stable causal association patterns, and clarifies the specific operations:

[0192] Calculate the frequency trend of each stable causal association pattern over a historical period (e.g., the frequency of Pattern_1 has gradually increased over the past 10 days).

[0193] Calculate the trend of stability score over time;

[0194] Does the time delay range of the calculation model change in a regular pattern over time (e.g., gradually shortening or lengthening)?

[0195] Finally, based on the above dynamic trend characteristics, the dynamic causal relationship pattern was further determined:

[0196] Example as follows:

[0197]

[0198] S104: Based on the dynamic causal relationship pattern and the real-time acquired downstream application performance monitoring data, generate an intelligent quantitative evaluation result of the impact of data quality on the actual performance of downstream applications.

[0199] In a specific implementation, the intelligent quantitative evaluation results of the impact of generated data quality on the actual performance of downstream applications include:

[0200] Real-time acquisition of performance monitoring data from downstream applications, and identification of abnormal fluctuations in performance metrics associated with data quality anomalies;

[0201] Specifically, in this embodiment, the performance monitoring data of downstream applications includes, but is not limited to, the following metrics:

[0202] (1) Application response latency: Records the latency of the application in responding to user requests or data queries, with the specific unit being milliseconds (ms).

[0203] (2) Throughput: Records the amount of requests or data that the application system can successfully process per unit of time (e.g., per second). The specific unit is requests / second or data volume / second.

[0204] (3) Success rate of call: The percentage of successful calls to the application service per unit time out of the total number of calls;

[0205] (4) Resource utilization rate: Record the percentage of CPU, memory, I / O and other resources used by downstream applications.

[0206] The above data is obtained by automatically collecting it through real-time monitoring probes or monitoring services (such as Prometheus, SkyWalking, and APM) deployed in downstream application systems, and transmitting it to the data processing platform in real time at minute or second intervals.

[0207] In practice, the steps for identifying abnormal fluctuations in performance indicators are fully disclosed as follows:

[0208] Perform streaming data processing on real-time monitoring data, including data cleaning and real-time anomaly detection;

[0209] The data cleaning process includes filtering out null values ​​and removing outliers (such as burst noise values) from real-time monitoring data to ensure the authenticity and stability of the data.

[0210] The real-time anomaly detection process employs a dynamic threshold detection method. Specifically, for each performance metric, the mean μ and standard deviation σ are calculated using a sliding window method (e.g., historical data from the most recent 10 minutes), and a dynamic threshold is set as follows:

[0211]

[0212] Where k is the anomaly detection sensitivity coefficient (e.g., k=3), and index values ​​exceeding this threshold range will be identified as abnormal fluctuation characteristics.

[0213] Further extraction of specific characteristics of the identified abnormal fluctuations in performance metrics, including:

[0214] Abnormal amplitude: The degree of deviation between the current performance indicator value and the dynamic threshold;

[0215] Abnormal duration: The duration during which the dynamic threshold is continuously exceeded;

[0216] Abnormal trend change rate: The abnormal fluctuation trend of performance indicators within a unit of time (such as increase or decrease).

[0217] For example, if the response latency of a downstream application reaches 250ms at a certain moment, which significantly exceeds the upper limit of the dynamic threshold of 200ms calculated in real time, then the abnormal amplitude is 50ms and the abnormal duration is 5 minutes.

[0218] The identified abnormal fluctuation characteristics of performance indicators are matched with the dynamic causal relationship pattern to determine the causal relationship pattern that is currently affecting the fluctuation of the current performance indicators.

[0219] In specific implementation, this embodiment fully discloses the real-time pattern matching method as follows:

[0220] For each identified abnormal fluctuation feature of a performance indicator, the corresponding real-time time series features are extracted, including: the start time of the abnormal fluctuation, the amplitude of the abnormal fluctuation and the amplitude of performance indicator changes recorded in the dynamic causal relationship pattern, and the actual time delay between the occurrence of the abnormal event and the fluctuation of the performance indicator.

[0221] The pattern similarity is calculated by using real-time features and pre-saved pattern features (such as time delay range, performance index change features, and stability score) in the dynamic causal relationship pattern generated in step S103.

[0222] In practice, the similarity calculation method uses weighted Euclidean distance, and the formula is as follows:

[0223]

[0224] In the formula: This indicates abnormal fluctuations in real-time performance metrics. Features of pre-saved dynamic causal relationship patterns; These represent the values ​​of the i-th dimension of the real-time feature and the pattern feature, respectively. The weights for the i-th dimension features are determined by historical data analysis, such as a time lag weight of 0.4, an anomaly amplitude weight of 0.4, and an anomaly duration weight of 0.2.

[0225] The calculated pattern similarities are sorted by magnitude, and the pattern with the highest similarity and meeting a preset matching threshold (e.g., similarity below the set threshold of 0.1) is determined as the current causal relationship pattern affecting performance metric fluctuations in real time; for example:

[0226]

[0227] Therefore, real-time matching determines that an anomaly with a latency fluctuation of 50ms and a time lag of 8min is matched with the first pre-stored pattern.

[0228] Based on the correlation characteristics of the causal relationship pattern in real time, an intelligent quantitative evaluation result is generated to assess the impact of data quality on the actual performance of downstream applications.

[0229] In its specific implementation, the intelligent quantitative evaluation method disclosed in this embodiment is fully disclosed as follows:

[0230] Based on the successfully matched dynamic causal association patterns, extract pattern association features (such as pattern stability score, pattern historical influence strength, and time lag range).

[0231] Based on the severity of the current abnormal fluctuations and the historical impact strength of the pattern, the real-time impact factor (IF) of the data quality anomalies on the actual performance of downstream applications is calculated using the following formula:

[0232]

[0233] In the formula: This represents the current real-time abnormal fluctuation amplitude; This represents the average amplitude of historical abnormal fluctuations in the model. Assign a score to the pattern stability. This represents the actual time delay of the current abnormal event; They are weighting factors (e.g.) ).

[0234] For example, assuming the real-time response latency fluctuation amplitude is 50ms, the historical average of the model is 45ms, the model stability score is 0.9, and the actual time delay is 8min, then the real-time impact factor (IF) is calculated as follows: ;

[0235] Based on the real-time impact factor (IF) value, the performance impact level is determined, and finally, an intelligent quantitative evaluation result is generated.

[0236] To further clarify, an example of the method for determining the impact level is provided:

[0237] If IF ≥ 0.8, it is defined as high impact, and a warning message is generated;

[0238] If 0.5 ≤ IF < 0.8, it is defined as medium impact, and a prompt message is generated;

[0239] If IF < 0.5, it is defined as low impact; information is recorded but no alarm is required.

[0240] The final output is an intelligent quantitative evaluation result, such as:

[0241]

[0242] Example 2

[0243] Please see Figure 2 As shown, this embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements any of the intelligent data quality assessment methods based on multimodal analysis provided by the above methods.

[0244] Since the electronic device described in this embodiment is the electronic device used to implement the intelligent data quality assessment method based on multimodal analysis in the embodiments of this application, those skilled in the art can understand the specific implementation method and various variations of the electronic device in this embodiment based on the intelligent data quality assessment method based on multimodal analysis described in the embodiments of this application. Therefore, how the electronic device implements the method in the embodiments of this application will not be described in detail here. Any electronic device used by those skilled in the art to implement the intelligent data quality assessment method based on multimodal analysis in the embodiments of this application falls within the scope of protection of this application.

[0245] Example 3

[0246] Please see Figure 3 As shown, this embodiment discloses a computer-readable storage medium, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, it implements any of the intelligent data quality assessment methods based on multimodal analysis provided by the above methods.

[0247] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters, weights, and thresholds in the formulas are set by those skilled in the art according to the actual situation.

[0248] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired or wireless network. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0249] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0250] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0251] In the several embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only one method, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0252] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0253] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0254] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0255] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A smart data quality assessment method based on multimodal analysis, characterized in that, include: Based on the data access logs of the cloud platform and the operation logs of downstream applications, the propagation path of data quality problems between heterogeneous systems is constructed; Based on the propagation path and the pre-constructed multimodal data quality problem impact knowledge graph, key related nodes in the propagation of data quality problems are identified; By combining the historical performance metrics data of the key related nodes and downstream applications, dynamic causal relationship patterns between the performance of key related nodes and downstream applications are explored; including: Obtain historical records of data quality anomaly events corresponding to key related nodes, as well as historical operational performance metrics data for downstream applications; Based on the temporal relationship between data quality anomaly events and performance index data, initial causal association candidate patterns are determined; The effectiveness of the initial causal association candidate patterns is evaluated using the causal pattern stability analysis method, and stable causal association patterns that meet the temporal consistency constraint are selected. Based on a stable causal relationship pattern, a dynamic causal relationship pattern between key related nodes and downstream application performance is generated. Based on the aforementioned dynamic causal relationship pattern and real-time acquired downstream application performance monitoring data, an intelligent quantitative evaluation result is generated regarding the impact of data quality on the actual performance of downstream applications; including: Real-time acquisition of performance monitoring data from downstream applications, and identification of abnormal fluctuations in performance metrics associated with data quality anomalies; The identified abnormal fluctuation characteristics of performance indicators are matched with the dynamic causal relationship pattern to determine the causal relationship pattern that is currently affecting the fluctuation of the current performance indicators. Based on the correlation characteristics of the causal relationship pattern in real time, an intelligent quantitative assessment result is generated to evaluate the impact of data quality on the actual performance of downstream applications.

2. The method according to claim 1, characterized in that, The propagation path of data quality issues between heterogeneous systems includes: Extract log entries that record data access anomalies from the cloud platform's data access logs, and identify the data resource identifier and access source system corresponding to the anomaly event; Extract log entries that record data processing anomalies from the downstream application's runtime logs, and identify the data resource identifiers and target application systems corresponding to the anomaly events; Based on the matching relationship of data resource identifiers and the time sequence of occurrence of abnormal events recorded in the logs, the cross-system propagation relationship between abnormal data access events and abnormal data processing events is determined. Based on the cross-system propagation relationship, a propagation path for data quality issues between heterogeneous systems is generated.

3. The method according to claim 2, characterized in that, The determination of the cross-system propagation relationship between data access anomaly events and data processing anomaly events includes: Based on the time range of the abnormal events in the logs, filter out data access abnormal events and data processing abnormal events that meet the time overlap condition; By utilizing system topology information, the selected events are matched with upstream and downstream nodes according to data resource identifiers to form cross-system event dependencies; Based on cross-system event dependencies, the cross-system propagation relationship between data access anomaly events and data processing anomaly events is generated.

4. The method according to claim 3, characterized in that, The multimodal data quality issues affect the knowledge graph construction method, including: Data access logs, downstream application operation logs, and system topology information are obtained from the cloud platform to extract system node characteristics and data quality anomaly event characteristics. Based on the temporal co-occurrence relationship of node characteristics during the occurrence of abnormal events and the interaction relationship between systems, the propagation correlation of data quality problems among nodes is determined; The data quality problem propagation correlation is integrated with the performance index correlation features contained in the system node features in a multimodal manner to generate a multimodal correlation structure between nodes; Based on the multimodal association structure between the nodes, a knowledge graph of the impact of multimodal data quality issues is generated.

5. The method according to claim 4, characterized in that, The multimodal integration of the data quality problem propagation correlation with the performance index correlation features contained in the system node characteristics includes: Based on the propagation correlation of data quality issues, the propagation time sequence features between nodes are extracted, and the change response features of node performance indicators are extracted from the system node features. The correlation strength between nodes is determined based on the matching relationship between propagation timing characteristics and the change response characteristics of node performance indicators; The multimodal association structure between nodes is generated based on the association strength between nodes.

6. The method according to claim 5, characterized in that, The effectiveness evaluation of the initial causal association candidate patterns includes: Based on the temporal characteristics of the initial causal relationship candidate patterns, we verify whether the temporal order constraints of events and performance indicators are met, and remove candidate patterns that do not meet the temporal constraints. By using independent historical data fragments, the recurrence of candidate patterns that meet the time series constraints is verified, and candidate patterns with recurrence stability are selected. Based on candidate patterns with repeatability stability, generate stable causal association patterns that satisfy temporal consistency constraints.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method according to any one of claims 1 to 6.

Images