Control method and control system of integrated data intelligence platform

Abstract

The application relates to the technical field of data intelligence, in particular to a control method and a control system of an integrated data intelligent platform. The application generates data link tracking parameters by configuring a data dependency relationship analysis mode and a target parameter, and obtains data node information to establish individualized dependency relationship parameters for each node; the system can analyze bidirectional data flow, trace data sources and generation processes upwards, and identify data flow and usage downwards; the system calculates and compares target parameters through integrity detection to obtain an integrity difference value; when the difference value exceeds a preset threshold value, a quality monitoring mechanism is triggered to timely discover and locate abnormalities; the system updates node dependency relationship parameters according to abnormal information to evaluate the influence range and degree of problem data; the accuracy and timeliness of data dependency relationship management are improved, accurate positioning and influence range evaluation of abnormalities are realized, and the dependency relationship analysis result can timely reflect the actual data flow situation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data intelligence, in particular to a control method and control system of an integrated data intelligence platform. BACKGROUND

[0002] With the growing demand for data governance, enterprises have increasingly high requirements for the management of data assets. Analysis and management of data dependency relationships have become one of the core tasks of data governance, which can help enterprises understand data flow processes, ensure data quality, and improve data use efficiency.

[0003] Currently, mainstream data governance platforms mainly build data dependency relationship graphs through metadata collection and rule configuration. These platforms automatically identify and track the dependency relationships between data by analyzing log information of data processing tasks, database table structures, and ETL job configurations, and other technical means.

[0004] However, the construction of data dependency relationships in the prior art is often static and cannot dynamically adapt to changes in the data flow process, lacks a quantitative evaluation mechanism for the completeness of data dependency relationships, and when data quality anomalies occur, it is difficult to accurately assess the scope and degree of impact. This situation needs to be further improved. SUMMARY

[0005] In order to solve the problem that the construction of data dependency relationships in the prior art is often static and cannot dynamically adapt to changes in the data flow process, lacks a quantitative evaluation mechanism for the completeness of data dependency relationships, and when data quality anomalies occur, it is difficult to accurately assess the scope and degree of impact, the present application provides a control method and control system of an integrated data intelligence platform, which adopts the following technical solutions:

[0006] In a first aspect, the present application provides a control method of an integrated data intelligence platform, comprising the following steps:

[0007] Obtain a data dependency relationship analysis mode and a target parameter, and obtain overall data link tracking parameters according to the data dependency relationship analysis mode and the target parameter;

[0008] Obtain data node information, and establish node dependency relationship parameters for each data node according to the data node information and the overall data link tracking parameters;

[0009] According to the node dependency relationship parameters and the overall data link tracking parameters, perform data flow analysis, including upward tracing of data sources and generation processes, and downward tracing of data flow and usage;

[0010] Perform integrity detection on the established data dependency relationship to obtain a link integrity detection parameter; compare the link integrity detection parameter with the target parameter to obtain a link integrity difference value;

[0011] If the link integrity difference value exceeds a preset threshold, trigger data link quality monitoring; and according to a data link quality monitoring result, obtain data link abnormal information;

[0012] According to the data link abnormal information, update node dependency relationship parameters, perform data impact assessment, and identify a problem data impact range and impact degree.

[0013] By adopting the above technical solution, the application first automatically generates tracking parameters of the overall data link by configuring a data dependency relationship analysis mode and combining target parameter setting; then, the system obtains detailed information of each data node and establishes individualized dependency relationship parameters for each node according to the tracking parameters; on this basis, the system can automatically perform bidirectional data flow analysis, which can trace the source and generation process of data upwards and identify the flow direction and use of data downwards; the system continuously performs integrity detection, obtains an integrity difference value by calculating a link integrity detection parameter and comparing it with a target parameter; when the difference value exceeds a preset threshold, the system will automatically trigger a quality monitoring mechanism to timely discover and locate abnormal conditions; finally, the system will update node dependency relationship parameters according to abnormal information and accurately identify the impact range and degree of problem data through data impact assessment; the accuracy and timeliness of data dependency relationship management are significantly improved; through the linkage mechanism of integrity detection and quality monitoring, accurate positioning of abnormalities and accurate assessment of the impact range are realized; through the dynamic updating mechanism of parameters, it is ensured that the dependency relationship analysis result can timely reflect the actual situation of data flow.

[0014] Optionally, the established data dependency relationship is subjected to integrity detection to obtain a link integrity detection parameter, specifically including the following steps:

[0015] According to the data node information and node dependency relationship parameters, calculate a dependency completeness index of the data node;

[0016] According to the dependency completeness index, calculate a data link coverage rate of each data node;

[0017] According to the data link coverage rate, analyze the difference degree between the current dependency relationship and the preset complete dependency relationship to obtain a dependency missing value of the current node;

[0018] According to the dependency missing value, estimate a comprehensive evaluation value of the integrity of the entire data link, perform integrity analysis on the data dependency relationship according to the comprehensive evaluation value of the integrity, and obtain the link integrity detection parameter.

[0019] By adopting the technical scheme, the application firstly constructs a calculation model by combining the basic information of the data nodes and the established node dependency relationship parameters, calculates the dependency completeness indexes of each data node, and further calculates the data link coverage of each data node based on the dependency completeness indexes, wherein the coverage represents the establishment degree of the dependency relationship between the nodes and other nodes; then, the system compares and analyzes the actually calculated data link coverage with the pre-set complete dependency relationship, obtains the dependency missing value of each node through the difference measurement algorithm, and the missing value directly reflects the incomplete degree of the current dependency relationship; the system estimates the completeness comprehensive evaluation value of the entire data link based on the dependency missing value of each node by using the weighted calculation method, and generates the link completeness detection parameter according to the completeness comprehensive evaluation value; by establishing the layered evaluation system of the completeness detection, it is ensured that the completeness detection result can fully reflect the condition of the entire data link.

[0020] Optionally, the method further comprises:

[0021] obtaining the dependency strength difference of the adjacent data nodes, and obtaining the data dependency degree and the data correlation degree corresponding to the data link coverage according to the dependency strength difference;

[0022] constructing the data dependency relationship topology graph corresponding to the dependency strength difference according to the data dependency degree and the data correlation degree;

[0023] analyzing the propagation characteristics of the data dependency relationship according to the data dependency relationship topology graph, and obtaining the dependency relationship completeness threshold of the current data environment;

[0024] when the actual dependency relationship of the data reaches the dependency relationship completeness threshold, outputting the dependency relationship optimization suggestion to the data link quality monitoring module.

[0025] By adopting the technical scheme, the application firstly obtains the difference in dependency strength between adjacent data nodes through an analysis algorithm, and calculates data dependency degree and data correlation degree corresponding to data link coverage based on the difference. The data dependency degree reflects the strength of direct dependency relationship between nodes, and the data correlation degree represents the degree of indirect influence between nodes. Then, the system takes the dependency degree and correlation degree information as a basis to construct a multi-dimensional data dependency relationship topology graph. The graph not only shows the connection relationship between nodes, but also intuitively expresses the difference in dependency strength through different connection strengths. Then, the system deeply analyzes the topology graph to study the propagation characteristics of data dependency relationship, including propagation path, propagation speed and propagation range, and calculates the dependency relationship integrity threshold in the current data environment according to the characteristics. Finally, when the system detects that the actual dependency relationship of data reaches the integrity threshold, the system automatically provides targeted optimization suggestions to the data link quality monitoring module, realizing the visual expression and quantitative analysis of dependency relationship.

[0026] Optionally, according to the data link quality evaluation result, data link abnormal information is obtained, specifically including the following steps:

[0027] According to the data link quality evaluation result, the data quality difference degree between adjacent data nodes is obtained.

[0028] According to the data quality difference degree, the data link is segmented and processed to generate data link segments of the same quality level.

[0029] The data link segment with a quality level lower than a preset quality threshold is marked as abnormal, and the abnormal link segment after marking is analyzed to obtain abnormal feature recognition parameters of the abnormal link segment.

[0030] According to the abnormal feature recognition parameters, the influence range corresponding to the abnormal link segment and the processing measures corresponding to each node are analyzed to obtain quality optimization processing parameters of the abnormal link segment.

[0031] By adopting the technical scheme, firstly, the system obtains the data quality difference between adjacent data nodes through comparative analysis algorithm based on the data link quality evaluation result, and the difference can accurately reflect the quality change of data in the transmission process; then, the system performs intelligent segmentation processing on the entire data link according to the calculated quality difference, and combines nodes with similar quality characteristics into data link segments with consistent quality levels, so that the abnormal positioning is more accurate; then, the system automatically marks the data link segments with a quality level lower than a preset quality threshold, and performs in-depth problem analysis on the marked quality abnormal link segments, obtains abnormal feature recognition parameters of the quality abnormal link segments through multi-dimensional feature extraction, and the parameters include key information such as type, degree and feature of the abnormality; finally, the system analyzes the influence range of the quality abnormal link segments based on the abnormal feature recognition parameters, formulates personalized processing measures for each affected node, and generates quality optimization processing parameters; through multi-dimensional analysis of abnormal features, the pertinence and effectiveness of the processing scheme are ensured.

[0032] Optionally, according to the data link abnormal information, the node dependency relationship parameters are updated, and data influence evaluation is performed, specifically including the following steps:

[0033] Obtain the abnormal feature type and abnormal influence degree of the current data node;

[0034] Analyze the influence area of the abnormal node on the associated node;

[0035] Evaluate the diffusion degree of abnormal data on link quality;

[0036] According to the abnormal feature type, abnormal influence degree, influence area and diffusion degree, an optimization suggestion of the dependency relationship is generated.

[0037] By adopting the technical scheme, firstly, the feature type and abnormal influence degree of the abnormal data node are obtained; then, the influence area of the abnormal node on the associated node is analyzed to identify the range of affected nodes; then, the diffusion degree of abnormal data on the entire link quality is evaluated, including diffusion speed and range; finally, the system generates an optimization suggestion of the dependency relationship based on the obtained information such as abnormal feature type, abnormal influence degree, influence area and diffusion degree; through the influence area analysis, the comprehensiveness of the influence evaluation is ensured; through the diffusion degree analysis, the evaluation result is more objective.

[0038] Optionally, the diffusion degree of abnormal data on link quality is evaluated, specifically including the following steps:

[0039] Identify the directly associated nodes of the abnormal node;

[0040] Evaluate the diffusion range of abnormal data;

[0041] calculating a quality fluctuation degree of each associated node;

[0042] generating a quality impact assessment report according to the direct associated node, the diffusion range and the quality fluctuation degree.

[0043] By adopting the technical solution, the application first identifies the associated nodes directly connected with the abnormal node through the dependency relationship network to determine the first layer impact range; then evaluates the diffusion range of the abnormal data in the entire network to track the transmission path of the abnormal impact; next, calculates the quality fluctuation degree of each affected node to quantify the impact caused by the abnormality; finally, the system automatically generates a quality impact assessment report containing detailed impact assessment information by comprehensively analyzing the aforementioned analysis results; the completeness of the impact analysis is ensured through the diffusion range evaluation; and the evaluation result is more quantitative through the quality fluctuation measurement.

[0044] In a second aspect, the application provides a control system of an integrated data intelligent platform, comprising:

[0045] a data acquisition module, configured to acquire a data dependency relationship analysis mode and a target parameter, acquire an overall data link tracking parameter according to the data dependency relationship analysis mode and the target parameter, acquire data node information, and establish a node dependency relationship parameter for each data node according to the data node information and the overall data link tracking parameter;

[0046] a data flow analysis module, configured to perform data flow analysis according to the node dependency relationship parameter and the overall data link tracking parameter, including upward source tracking of data and generation process and downward tracking of data flow and use;

[0047] a completeness detection module, configured to perform completeness detection on the established data dependency relationship to acquire a link completeness detection parameter, compare the link completeness detection parameter with the target parameter to obtain a link completeness difference value;

[0048] a quality monitoring module, configured to trigger data link quality monitoring when the link completeness difference value exceeds a preset threshold, and acquire data link abnormal information according to a data link quality monitoring result;

[0049] an impact assessment module, configured to update the node dependency relationship parameter according to the data link abnormal information, perform data impact assessment, and identify a problem data impact range and impact degree.

[0050] Optionally, the completeness detection module comprises:

[0051] a completeness calculation unit configured to calculate a dependency completeness index of the data node according to the data node information and the node dependency parameter;

[0052] a coverage calculation unit configured to calculate a data link coverage of each data node according to the dependency completeness index;

[0053] a missing value analysis unit configured to analyze a difference between a current dependency relationship and a preset complete dependency relationship according to the data link coverage, to obtain a dependency missing value of the current node;

[0054] a comprehensive evaluation unit configured to estimate a completeness comprehensive evaluation value of the entire data link according to the dependency missing value, to perform completeness analysis on the data dependency relationship according to the completeness comprehensive evaluation value, and to obtain the link completeness detection parameter.

[0055] Optionally, the quality monitoring module further comprises:

[0056] a dependency strength analysis unit configured to obtain a dependency strength difference between adjacent data nodes, to obtain a data dependency degree and a data correlation degree corresponding to the data link coverage according to the dependency strength difference;

[0057] a topology graph construction unit configured to construct a data dependency relationship topology graph corresponding to the dependency strength difference according to the data dependency degree and the data correlation degree;

[0058] a propagation feature analysis unit configured to analyze a propagation feature of the data dependency relationship according to the data dependency relationship topology graph, to obtain a dependency relationship completeness threshold of a current data environment;

[0059] an optimization suggestion generation unit configured to output a dependency relationship optimization suggestion to the data link quality monitoring module when an actual dependency relationship of the data reaches the dependency relationship completeness threshold.

[0060] Optionally, the quality monitoring module comprises:

[0061] a quality difference analysis unit configured to obtain a data quality difference degree between adjacent data nodes according to the data link quality evaluation result;

[0062] a link segmentation processing unit configured to perform segmentation processing on the data link according to the data quality difference degree, to generate a data link segment of a same quality level;

[0063] an abnormality identification unit configured to mark a data link segment of a quality level lower than a preset quality threshold as abnormal, to perform problem analysis on the marked quality abnormal link segment, and to obtain an abnormal feature identification parameter of the quality abnormal link segment;

[0064] A processing parameter generation unit is configured to obtain quality optimization processing parameters of the quality abnormal link segment according to the abnormal feature identification parameters, analyze an influence range corresponding to the quality abnormal link segment and processing measures corresponding to each node, and obtain the quality optimization processing parameters of the quality abnormal link segment.

[0065] In summary, the present application includes at least one of the following beneficial technical effects:

[0066] 1. The present application generates data link tracking parameters by configuring data dependency analysis mode and target parameters, and obtains data node information to establish personalized dependency relationship parameters for each node; the system can analyze bidirectional data flow, trace data sources and generation process upwards, and identify data flow direction and usage downwards; the integrity difference value is obtained by calculating and comparing target parameters through integrity detection; when the difference value exceeds the preset threshold, the quality monitoring mechanism is triggered to timely discover and locate abnormalities; the node dependency relationship parameters are updated according to the abnormal information to evaluate the influence range and degree of problem data; the accuracy and timeliness of data dependency relationship management are improved, accurate positioning and influence range evaluation of abnormalities are realized, and the dependency relationship analysis result is ensured to reflect the actual data flow situation in time;

[0067] 2. The present application constructs a calculation model based on data node information and dependency relationship parameters, calculates the dependency integrity index of each node, and further calculates the data link coverage rate for representing the establishment degree of dependency relationship between nodes; the system compares the actual coverage rate with the preset complete dependency relationship, obtains the node dependency missing value through the difference measurement algorithm, and directly reflects the incomplete degree of dependency relationship; based on the dependency missing value of each node, the weighted calculation method is used to estimate the integrity comprehensive evaluation value of the data link to generate the link integrity detection parameter, and ensure that the detection result comprehensively reflects the data link condition;

[0068] 3. The present application obtains the dependency strength difference between adjacent data nodes through an analysis algorithm, calculates the data dependency degree and correlation degree corresponding to the data link coverage rate, and reflects the direct dependency strength and indirect influence degree between nodes; the system constructs a multi-dimensional data dependency relationship topology graph based on these information to directly display the node connection relationship and dependency strength difference; the dependency relationship integrity threshold is calculated by analyzing the propagation path, speed and range of the topology graph; when the actual dependency relationship reaches the threshold, the optimization suggestion is provided to the quality monitoring module to realize the visual expression and quantitative analysis of the dependency relationship. BRIEF DESCRIPTION OF DRAWINGS

[0069] Figure 1 is a flowchart of the control method of the integrated data intelligent platform of the embodiment of the present application;

[0070] Figure 2 is a flowchart of step S400 in the control method of the integrated data intelligent platform of the embodiment of the present applicationFigure One ;

[0071] Figure 3 is a flowchart of step S400 in the control method of the integrated data intelligent platform of the embodiment of the present application Figure Two ;

[0072] Figure 4 is a flowchart of step S500 in the control method of the integrated data intelligent platform of the embodiment of the present application

[0073] Figure 5 is a flowchart of step S600 in the control method of the integrated data intelligent platform of the embodiment of the present application

[0074] Figure 6 is a flowchart of step S630 in the control method of the integrated data intelligent platform of the embodiment of the present application

[0075] Figure 7 is a module schematic diagram of the control system of the integrated data intelligent platform of the embodiment of the present application DETAILED DESCRIPTION

[0076] The terms used in the following embodiments of the present application are only for the purpose of describing the specific embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims of the present application, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "and / or" as used herein refers to any or all possible combinations of one or more of the associated listed items.

[0077] Hereinafter, the terms "first", "second", etc. are used only for the purpose of description and should not be understood as implying or suggesting relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0078] The embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0079] In a first aspect, the present application provides a control method of an integrated data intelligent platform, referring to Figure 1 , comprising the following steps:

[0080] S100, acquiring a data dependency analysis mode and a target parameter, and acquiring an overall data link tracking parameter according to the data dependency analysis mode and the target parameter.

[0081] In this embodiment, the data dependency analysis mode refers to the identification method of the dependency relationship between data nodes, including data table blood relationship analysis, field-level mapping analysis, data call relationship analysis, and business process association analysis. The target parameters include node coverage target value, link connectivity target value, and data transmission completeness target value. The overall data link tracking parameter is the key configuration information for guiding subsequent dependency analysis, including node identification rules, dependency judgment criteria, and tracking depth limit.

[0082] Specifically, the system first selects a suitable dependency relationship analysis mode through a pre-established analysis mode mapping table. The mapping table stores the applicable analysis mode combination in different data scenarios, for example, in the data warehouse scenario, the combination mode of data table blood relationship analysis and field-level mapping analysis is selected. Then, the system sets the target parameters based on historical experience values, such as setting the node coverage target value to 95%, the link connectivity target value to 98%, and the data transmission completeness target value to 90%.

[0083] S200, obtain data node information, and based on the data node information and the overall data link tracking parameter, establish node dependency relationship parameters for each data node.

[0084] In this embodiment, the data node information includes node identification information, node type information, and node content information. The node dependency relationship parameters include the direct association strength between nodes, data transmission frequency, and data change impact degree.

[0085] Specifically, the system obtains node identification information through the configured data dictionary, reads node type information from the metadata management library, and obtains node content information through the data acquisition interface. Based on the obtained node information, the system uses the pre-constructed feature extraction rules to generate node feature vectors. Combined with the overall data link tracking parameter, the system calculates the dependency relationship parameters for each node through the rule calculation engine, including the association strength score of 0.8 with adjacent nodes, the data transmission frequency of once every 5 minutes, and the data change impact degree of moderate impact.

[0086] S300, according to the node dependency relationship parameters and the overall data link tracking parameter, perform data flow analysis, including upward traceability tracking data source and generation process, and downward tracking to identify data flow and usage.

[0087] In this embodiment, the data flow analysis includes upstream traceability analysis and downstream tracking analysis. The upstream traceability analysis is responsible for identifying the generation source and processing process of data, and the downstream tracking analysis is responsible for tracking the usage scenario and distribution of data.

[0088] Specifically, the system first constructs a data flow index table to record the data transmission relationship between nodes. In upstream traceability analysis, the system traces the data source layer by layer upwards by querying the index table combined with node dependency parameters. In downstream tracking analysis, the system tracks the data flow downwards in the same way to identify the use of data. The system saves the complete data flow recorded in the flow analysis result table.

[0089] S400, integrity detection is performed on the established data dependency relationship, and a link integrity detection parameter is obtained; the link integrity detection parameter is compared with a target parameter to obtain a link integrity difference value.

[0090] In this embodiment, the link integrity detection parameter reflects the completeness of the data dependency relationship, including node coverage completeness, link connectivity completeness, and data transmission completeness. The link integrity difference value represents the gap between the actual integrity and the target parameter.

[0091] Specifically, the system performs detection through the integrity detection rule library. First, the node coverage completeness is calculated by matching the ratio of the actual number of nodes to the expected number of nodes through the node mapping relationship table. Then, the link connectivity completeness is calculated by verifying the availability of data transmission channels between nodes using the connectivity detection module. Next, the data transmission completeness is calculated by using data sampling to count the data consistency in the transmission process. The system compares these detection parameters with the target parameters to obtain a completeness difference value of 7%.

[0092] S500, if the link integrity difference value exceeds a preset threshold, data link quality monitoring is triggered; and data link abnormal information is obtained according to the data link quality monitoring result.

[0093] In this embodiment, data link quality monitoring includes node quality monitoring, connectivity quality monitoring, and transmission quality monitoring. Data link abnormal information records specific problems and their characteristic descriptions found in the monitoring process.

[0094] Specifically, when the completeness difference value exceeds the preset threshold of 5%, the system starts the quality monitoring process. The node quality monitoring verifies the existence and normality of node data through the node state check table, such as finding missing node data, format errors; the connectivity quality monitoring checks the connection state between nodes based on the link detection module, such as finding data transmission interruption, transmission delay; the transmission quality monitoring verifies the accuracy of data transmission using data comparison rules, such as finding data inconsistency, data duplication. For example, the system finds that the node coverage completeness decreases due to abnormal data format of the payment node in a certain order processing link, the link connectivity completeness decreases due to the disconnection between the order node and the payment node, and the transmission completeness decreases due to the field missing in the data transmission process. These specific abnormal information is recorded in the monitoring result library.

[0095] S600, updating the node dependency parameter according to the data link exception information, performing data impact assessment, and identifying the problem data impact range and impact degree.

[0096] In this embodiment, the data impact assessment includes direct impact assessment and cascading impact assessment. The direct impact assessment is performed on the adjacent nodes of the abnormal node, and the cascading impact assessment analyzes all the nodes affected on the abnormal propagation path.

[0097] In one embodiment, referring to Figure 2 In step S400, the integrity of the established data dependency is detected, and link integrity detection parameters are obtained, including the following steps:

[0098] S410, calculating the dependency integrity index of the data node according to the data node information and the node dependency parameter.

[0099] In this embodiment, the dependency integrity index includes data integrity rate, update timeliness and data quality score of the data node. The data integrity rate reflects the completeness of the node data field, the update timeliness indicates the effectiveness of data update, and the data quality score indicates the accuracy of data content.

[0100] Specifically, the system calculates the dependency integrity index through the node evaluation rule library. The data integrity rate is verified by a pre-set field check table, and the ratio of the actual field number to the standard field number is taken as the integrity rate; the update timeliness is calculated based on a timestamp comparison table, and the compliance of the data update time with the standard update period is recorded; the data quality score is scored by a quality scorecard, including field format specification, numerical interval rationality, and key field uniqueness. The system takes the weighted result of the three indexes as the dependency integrity index of the node.

[0101] S420, calculating the data link coverage rate of each data node according to the dependency integrity index.

[0102] In this embodiment, the data link coverage rate refers to the establishment degree of the dependency relationship between the data node and its associated nodes. It includes three dimensions: upstream node coverage, downstream node coverage and associated node coverage.

[0103] Specifically, the system calculates the data link coverage based on the dependency completeness index. First, the standard dependency relationship list is obtained through the node relationship mapping table, which records all the dependency relationships that the node should establish. Then, by comparing the actual established dependency relationships, the coverage of three dimensions is calculated: the upstream node coverage is calculated by tracing the source analysis table, which records the connection integrity with the upstream data source; the downstream node coverage is calculated by the application call record table, which records the coverage of the downstream application; the associated node coverage is calculated based on the business mapping table, which calculates the connection degree of the business related nodes.

[0104] S430, according to the data link coverage, analyze the difference between the current dependency relationship and the preset complete dependency relationship, and obtain the dependency missing value of the current node.

[0105] In this embodiment, the dependency missing value represents the degree of incompleteness of the current node dependency relationship. The preset complete dependency relationship is stored in the dependency relationship standard library, which includes three benchmark indicators of the standard connection number, connection mode and connection strength between nodes. The difference degree is measured by the deviation of the actual value and the standard value.

[0106] Specifically, the system calculates the dependency missing value through the difference analysis module. First, the preset complete dependency relationship benchmark value is read from the dependency relationship standard library, which is determined based on the business scenario and the importance of data. Then, the data link coverage is compared with the benchmark value: the connection number difference is calculated through the node connection statistics table, which records the difference between the actual connection number and the standard connection number; the connection mode difference is identified through the connection type comparison table, which marks the connection type that does not meet the standard; the connection strength difference is calculated based on the weight matrix, which analyzes the deviation between the actual connection strength and the standard strength. The system calculates the dependency missing value by integrating the three difference values.

[0107] S440, according to the dependency missing value, estimate the integrity comprehensive evaluation value of the entire data link, and according to the integrity comprehensive evaluation value, analyze the integrity of the data dependency relationship, and obtain the link integrity detection parameter.

[0108] In this embodiment, the integrity comprehensive evaluation value is a whole measurement indicator of the integrity of the entire data link. The link integrity detection parameter includes the specific detection results of three dimensions of node coverage completeness, link connectivity completeness and data transmission completeness.

[0109] Specifically, the system first establishes an evaluation weight table, and assigns different weight coefficients according to the business importance of the nodes. The missing values of the dependency of each node are summarized through weighted calculation to obtain a comprehensive evaluation value of the integrity of the data link. The system uses a hierarchical evaluation mechanism for integrity analysis: the first layer checks the data state of each node through a node state table; the second layer verifies the connection state between nodes using a connectivity detection table; and the third layer confirms the accuracy of data transmission based on a data consistency verification table. Finally, a link integrity detection parameter containing these three dimensions is generated.

[0110] In one embodiment, with reference to Figure 3 , the method further comprises:

[0111] S450, obtaining the dependency strength difference between adjacent data nodes, and obtaining the data dependency degree and the data correlation degree corresponding to the data link coverage rate according to the dependency strength difference.

[0112] In this embodiment, the dependency strength difference refers to the difference in data interaction capability between adjacent nodes. The data dependency degree represents the strength of the direct dependency relationship between nodes, which is calculated based on three indicators: data call frequency, data size, and call stability. The data correlation degree represents the degree of indirect influence between nodes, which is calculated based on three indicators: business correlation, data similarity, and transmission path length.

[0113] Specifically, the system obtains the dependency strength difference through the strength calculation module. First, a node interaction record table is established to record the data exchange between nodes: the data call frequency is calculated through the interface call log, the data size is obtained through the data flow statistics table, and the call stability is calculated through the fault record table. Then, the data dependency degree is calculated based on these records. At the same time, an association analysis table is established to calculate the data correlation degree: the business correlation is determined through the business rule configuration table, the data similarity is compared through the field mapping table, and the transmission path length is obtained through the path tracking table.

[0114] S460, according to the data dependency degree and the data correlation degree, constructing a data dependency relationship topology graph corresponding to the dependency strength difference.

[0115] In this embodiment, the data dependency relationship topology graph is a visual dependency relationship display structure, where nodes represent data processing units, lines represent dependency relationships, line thickness represents dependency strength, and line type represents dependency direction.

[0116] Specifically, the system generates a topology graph through a graph construction module. First, a node layout table is established to specify the position distribution of nodes in the graph, with core nodes in the center and dependent nodes distributed by level. Then, the connection drawing rule table is used to determine the line properties: thick solid lines are used for a dependency degree greater than 0.8, thin solid lines are used for a dependency degree between 0.5 and 0.8, and dashed lines are used for a dependency degree less than 0.5. The correlation degree is identified by different colors: red for high correlation degree, yellow for medium correlation degree, and green for low correlation degree. The system applies these rules to the graph generator to build a complete dependency relationship topology graph.

[0117] S470, according to the data dependency relationship topology graph, the propagation characteristics of the data dependency relationship are analyzed to obtain a dependency relationship integrity threshold of the current data environment.

[0118] In this embodiment, the propagation characteristics include three dimensions of propagation path length, propagation breadth, and propagation depth. The dependency relationship integrity threshold is a standard line for measuring the establishment degree of the dependency relationship, which is composed of a basic threshold and a dynamic adjustment factor. The basic threshold is set based on historical experience, and the dynamic adjustment factor is calculated according to the characteristics of the current data environment.

[0119] Specifically, the system processes the data dependency relationship topology graph through a feature analysis module. First, a propagation characteristic analysis table is established: the propagation path length is counted by a path counter to obtain the longest dependency chain, the propagation breadth is calculated by a node distribution matrix to obtain the horizontal coverage range, and the propagation depth is recorded by a level statistics table to obtain the vertical influence level.

[0120] S480, when the actual dependency relationship of the data reaches the dependency relationship integrity threshold, a dependency relationship optimization suggestion is output to the data link quality monitoring module.

[0121] In this embodiment, the dependency relationship optimization suggestion includes three parts of optimization target, optimization scheme, and implementation suggestion. The optimization target indicates the specific indicators that need to be improved, the optimization scheme provides feasible improvement measures, and the implementation suggestion gives specific execution steps.

[0122] In one embodiment, referring to Figure 4 , in step S500, data link abnormal information is obtained according to the data link quality evaluation result, specifically including the following steps:

[0123] S510, according to the data link quality evaluation result, the data quality difference degree between adjacent data nodes is obtained.

[0124] In this embodiment, the data quality difference degree refers to the degree of difference in data quality between adjacent nodes. The quality difference includes three dimensions of data specification difference, data integrity difference, and data consistency difference.

[0125] S520, segment the data link according to the data quality difference, and generate data link segments with the same quality level.

[0126] In this embodiment, the data link segment refers to a continuous node sequence with similar quality characteristics. The quality level is divided into four levels: excellent, good, general, and poor, each level having clear determination criteria and grading thresholds.

[0127] Specifically, the system divides the link through the segment processing module. First, a quality grading table is established, and determination rules for different levels are set. Then, the quality is evaluated segment by segment along the data flow direction, and nodes with similar quality difference are combined into link segments.

[0128] S530, mark the data link segment with a quality level lower than the preset quality threshold as abnormal, and analyze the marked quality abnormal link segment to obtain abnormal feature identification parameters of the quality abnormal link segment.

[0129] In this embodiment, the abnormal marking is a special identification of the link segment with a quality level lower than the preset threshold. The abnormal feature identification parameters include three indicators: abnormal type identification, abnormal degree score, and abnormal duration. The preset quality threshold is determined based on the importance of the business and the sensitivity of the data.

[0130] Specifically, the system marks and analyzes through the abnormal analysis module. First, an abnormal determination table is established, and the link segment with a quality level of "poor" or "general" is marked as an abnormal segment. Then, the problem feature library is analyzed: the abnormal type is identified through the feature matching table, such as data fault, data tilt, and data delay; the abnormal degree is calculated through the scoring rule table, and the impact degree is quantified as 1-10 points; and the abnormal duration is obtained through the time window statistics.

[0131] S540, according to the abnormal feature identification parameters, analyze the impact range of the quality abnormal link segment and the processing measures corresponding to each node, and obtain the quality optimization processing parameters of the quality abnormal link segment.

[0132] In one embodiment, referring to Figure 5 , in step S600, according to the data link abnormal information, update the node dependency relationship parameters, and perform data impact evaluation, which specifically includes the following steps:

[0133] S610, obtain the abnormal feature type and abnormal impact degree of the current data node.

[0134] S620, analyze the influence area of the abnormal node on the associated nodes.

[0135] In this embodiment, the influence area refers to the range of all related nodes that may be affected by the abnormal node. The related nodes include direct related nodes and indirect related nodes. The direct related nodes are directly connected through data interaction, and the indirect related nodes are associated through business logic.

[0136] Specifically, the system determines the influence range through the association analysis module. First, a node association table is established to record the association relationship between nodes: direct association is identified through a data flow diagram, and nodes that have data interaction with the abnormal node are found; indirect association is determined through a business rule table, and nodes that are dependent on each other in business are identified. Then, the possibility of each associated node being affected is calculated: the influence probability of the direct associated node is evaluated through a dependency strength matrix, and the influence degree of the indirect associated node is evaluated through a business importance table.

[0137] S630, evaluate the diffusion degree of abnormal data on link quality.

[0138] S640, generate optimization suggestions for dependency relationship according to abnormal feature type, abnormal influence degree, influence area and diffusion degree.

[0139] In one embodiment, with reference to Figure 6 In step S630, the diffusion degree of abnormal data on link quality is evaluated, which specifically includes the following steps:

[0140] S631, identify direct associated nodes of the abnormal node.

[0141] In this embodiment, the direct associated node refers to a node that has data interaction with the abnormal node. Data interaction includes three types of data input relationship, data output relationship and data sharing relationship. Node identification is based on two dimensions of data flow and business call.

[0142] S632, evaluate the diffusion range of abnormal data.

[0143] In this embodiment, the diffusion range describes the spatial distribution characteristics of abnormal influence. It includes two dimensions of horizontal diffusion range and vertical diffusion range. Horizontal diffusion represents the influence coverage of nodes in the same level, and vertical diffusion represents the influence transmission across nodes in different levels.

[0144] S633, calculate the quality fluctuation degree of each associated node.

[0145] In this embodiment, the quality fluctuation degree represents the change amplitude of data quality of the associated node. It includes three indicators of data accuracy fluctuation, data completeness fluctuation and processing efficiency fluctuation. Fluctuation calculation is based on data comparison before and after the occurrence of the abnormality, and the fluctuation intensity is measured by standard deviation and coefficient of variation.

[0146] Specifically, the system analyzes the quality change through the fluctuation calculation module. First, a quality monitoring table is established to record the quality indicators of the associated nodes: the data accuracy fluctuation is calculated through the correctness check table to record the error rate change; the data integrity fluctuation is calculated through the integrity check table to track the data loss rate change; and the processing efficiency fluctuation is measured through the performance monitoring table to calculate the processing delay change.

[0147] S634, generating a quality impact assessment report according to the directly associated nodes, the diffusion range, and the quality fluctuation degree.

[0148] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0149] In a second aspect, the present application provides a control system of an integrated data intelligent platform. The control system of the integrated data intelligent platform of the present application will be described below in combination with the control method of the integrated data intelligent platform.

[0150] Referring to Figure 7 A control system of an integrated data intelligent platform, comprising:

[0151] A data acquisition module is configured to acquire a data dependency analysis mode and a target parameter, acquire an overall data link tracking parameter according to the data dependency analysis mode and the target parameter, acquire data node information, and establish a node dependency relationship parameter for each data node according to the data node information and the overall data link tracking parameter.

[0152] A data flow analysis module is configured to perform data flow analysis according to the node dependency relationship parameter and the overall data link tracking parameter, including upward tracing of data sources and generation processes, and downward tracing of data flow directions and usage.

[0153] An integrity detection module is configured to perform integrity detection on the established data dependency relationship, acquire a link integrity detection parameter, compare the link integrity detection parameter with the target parameter, and obtain a link integrity difference value.

[0154] A quality monitoring module is configured to trigger data link quality monitoring when the link integrity difference value exceeds a preset threshold, and acquire data link abnormal information according to a data link quality monitoring result.

[0155] An impact assessment module is configured to update the node dependency relationship parameter according to the data link abnormal information, perform data impact assessment, and identify a problem data impact range and an impact degree.

[0156] In one embodiment, the integrity detection module comprises:

[0157] a completeness calculation unit, configured to calculate a dependency completeness index of the data node according to the data node information and the node dependency parameter;

[0158] a coverage calculation unit, configured to calculate a data link coverage of each data node according to the dependency completeness index;

[0159] a missing value analysis unit, configured to analyze a difference between a current dependency relationship and a preset complete dependency relationship according to the data link coverage, to obtain a dependency missing value of the current node;

[0160] a comprehensive evaluation unit, configured to estimate a completeness comprehensive evaluation value of the entire data link according to the dependency missing value, and perform completeness analysis on the data dependency relationship according to the completeness comprehensive evaluation value, to obtain a link completeness detection parameter.

[0161] In an embodiment, the quality monitoring module further comprises:

[0162] a dependency strength analysis unit, configured to obtain a dependency strength difference between adjacent data nodes, and obtain a data dependency degree and a data correlation degree corresponding to the data link coverage according to the dependency strength difference;

[0163] a topology graph construction unit, configured to construct a data dependency relationship topology graph corresponding to the dependency strength difference according to the data dependency degree and the data correlation degree;

[0164] a propagation feature analysis unit, configured to analyze a propagation feature of the data dependency relationship according to the data dependency relationship topology graph, to obtain a dependency relationship completeness threshold of a current data environment;

[0165] an optimization suggestion generation unit, configured to output a dependency relationship optimization suggestion to the data link quality monitoring module when an actual dependency relationship of the data reaches the dependency relationship completeness threshold.

[0166] In an embodiment, the quality monitoring module comprises:

[0167] a quality difference analysis unit, configured to obtain a data quality difference degree between adjacent data nodes according to a data link quality evaluation result;

[0168] a link segmentation processing unit, configured to perform segmentation processing on the data link according to the data quality difference degree, to generate a data link segment of a same quality level;

[0169] an abnormality identification unit, configured to mark a data link segment of a quality level lower than a preset quality threshold as abnormal, and perform problem analysis on the marked quality abnormal link segment, to obtain an abnormal feature identification parameter of the quality abnormal link segment;

[0170] The processing parameter generation unit is configured to obtain quality optimization processing parameters of the quality abnormal link segment according to the abnormal feature identification parameter, the influence range corresponding to the quality abnormal link segment, and the processing measure corresponding to each node.

[0171] In one embodiment, the influence evaluation module comprises:

[0172] The abnormal feature analysis unit is configured to obtain an abnormal feature type and an abnormal influence degree of the current data node.

[0173] The influence area analysis unit is configured to analyze an influence area of the abnormal node on the associated node.

[0174] The diffusion degree evaluation unit is configured to evaluate a diffusion degree of the abnormal data on the link quality.

[0175] The optimization suggestion generation unit is configured to generate an optimization suggestion of the dependency relationship according to the abnormal feature type, the abnormal influence degree, the influence area, and the diffusion degree.

[0176] In one embodiment, the diffusion degree evaluation unit comprises:

[0177] The associated node identification module is configured to identify a direct associated node of the abnormal node.

[0178] The diffusion range evaluation module is configured to evaluate a diffusion range of the abnormal data.

[0179] The fluctuation degree calculation module is configured to calculate a quality fluctuation degree of each associated node.

[0180] The evaluation report generation module is configured to generate a quality influence evaluation report according to the direct associated node, the diffusion range, and the quality fluctuation degree.

[0181] The above are preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be covered within the protection scope of the present application.

Claims

1. A control method for an integrated data intelligence platform, characterized in that, Includes the following steps: Obtain the data dependency analysis mode and target parameters, and obtain the overall data link tracing parameters based on the data dependency analysis mode and target parameters; Obtain data node information, and establish node dependency parameters for each data node based on the data node information and the overall data link tracing parameters; Based on the node dependency parameters and the overall data link tracing parameters, data flow analysis is performed, including tracing the data source and generation process upstream and identifying the data flow and usage downstream. Integrity checks are performed on the established data dependencies to obtain link integrity check parameters; the link integrity check parameters are compared with the target parameters to obtain the link integrity difference value; If the difference in link integrity exceeds a preset threshold, data link quality monitoring is triggered; based on the data link quality monitoring results, data link anomaly information is obtained. Based on the data link anomaly information, update the node dependency parameters, conduct a data impact assessment, and identify the scope and degree of impact of the problematic data.

2. The control method for the integrated data intelligence platform according to claim 1, characterized in that, Perform integrity checks on the established data dependencies and obtain link integrity check parameters. The specific steps include the following: Based on the data node information and node dependency parameters, calculate the dependency integrity index of the data node; Calculate the data link coverage of each data node based on the aforementioned dependency integrity metric; Based on the data link coverage, analyze the degree of difference between the current dependency relationship and the preset complete dependency relationship to obtain the dependency missing value of the current node; The integrity assessment value of the entire data link is estimated based on the missing dependency value. The integrity assessment value is then used to perform integrity analysis on the data dependencies to obtain the link integrity detection parameters.

3. The control method for the integrated data intelligence platform according to claim 2, characterized in that, The method further includes: Obtain the difference in dependency strength between adjacent data nodes, and obtain the data dependency and data correlation corresponding to the data link coverage based on the difference in dependency strength; Based on the data dependency degree and the data correlation degree, construct a data dependency relationship topology graph corresponding to the difference in dependency strength; Based on the data dependency topology graph, the propagation characteristics of the data dependencies are analyzed to obtain the dependency integrity threshold of the current data environment. When the actual data dependencies reach the dependency integrity threshold, dependency optimization suggestions are output to the data link quality monitoring module.

4. The control method for the integrated data intelligence platform according to claim 1, characterized in that, Based on the data link quality assessment results, data link anomaly information is obtained, specifically including the following steps: Based on the data link quality assessment results, the data quality difference between adjacent data nodes is obtained; The data link is segmented according to the data quality difference to generate data link segments of the same quality level. Data link segments with quality levels below a preset quality threshold are marked as abnormal, and the marked abnormal link segments are analyzed to obtain the abnormal feature identification parameters of the abnormal link segments. Based on the abnormal feature identification parameters, the impact range of the abnormal link segment and the corresponding handling measures for each node are analyzed to obtain the quality optimization handling parameters for the abnormal link segment.

5. The control method for the integrated data intelligence platform according to claim 1, characterized in that, Based on the data link anomaly information, update the node dependency parameters and perform a data impact assessment, specifically including the following steps: Obtain the anomaly type and impact level of the current data node; Analyze the area of ​​influence of abnormal nodes on related nodes; Assess the extent to which anomalous data spreads to link quality; Based on the anomaly type, the degree of anomaly impact, the affected area, and the degree of diffusion, optimization suggestions for dependencies are generated.

6. The control method for the integrated data intelligence platform according to claim 5, characterized in that, Assessing the extent to which anomalous data affects link quality involves the following steps: Identify the direct related nodes of the abnormal nodes; Assess the extent of the spread of anomalous data; Calculate the quality fluctuation level of each associated node; A quality impact assessment report is generated based on the directly related nodes, the diffusion range, and the degree of quality fluctuation.

7. A control system for an integrated data intelligence platform, characterized in that, include: The data acquisition module is used to obtain the data dependency analysis mode and target parameters, and to obtain the overall data link tracing parameters based on the data dependency analysis mode and the target parameters. Obtain data node information, and establish node dependency parameters for each data node based on the data node information and the overall data link tracing parameters; The data flow analysis module is used to perform data flow analysis based on the node dependency parameters and the overall data link tracing parameters, including tracing the data source and generation process upstream and identifying the data flow and usage downstream. The integrity detection module is used to perform integrity detection on the established data dependencies, obtain link integrity detection parameters, and compare the link integrity detection parameters with the target parameters to obtain the link integrity difference value. The quality monitoring module is used to trigger data link quality monitoring when the link integrity difference value exceeds a preset threshold, and to obtain data link anomaly information based on the data link quality monitoring results. The impact assessment module is used to update node dependency parameters based on the data link anomaly information, perform data impact assessment, and identify the scope and degree of impact of the problematic data.

8. The control system of the integrated data intelligence platform according to claim 7, characterized in that, The integrity detection module includes: The integrity calculation unit is used to calculate the dependency integrity index of the data node based on the data node information and node dependency parameters. The coverage calculation unit is used to calculate the data link coverage of each data node based on the dependency integrity index. The missing value analysis unit is used to analyze the degree of difference between the current dependency relationship and the preset complete dependency relationship based on the data link coverage, and obtain the dependency missing value of the current node. The comprehensive evaluation unit is used to estimate the comprehensive integrity evaluation value of the entire data link based on the dependency missing value, perform integrity analysis on the data dependency relationship based on the comprehensive integrity evaluation value, and obtain the link integrity detection parameters.

9. The control system of the integrated data intelligence platform according to claim 7, characterized in that, Also includes: The dependency strength analysis unit is used to obtain the dependency strength difference between adjacent data nodes, and to obtain the data dependency and data correlation corresponding to the data link coverage based on the dependency strength difference; The topology graph construction unit is used to construct a data dependency relationship topology graph corresponding to the difference in dependency strength based on the data dependency degree and the data correlation degree. The propagation feature analysis unit is used to analyze the propagation features of the data dependency relationship based on the data dependency relationship topology graph, and obtain the dependency relationship integrity threshold of the current data environment. An optimization suggestion generation unit is used to output dependency optimization suggestions to the data link quality monitoring module when the actual dependency relationship of the data reaches the dependency integrity threshold.

10. The control system of the integrated data intelligence platform according to claim 7, characterized in that, The quality monitoring module includes: The quality difference analysis unit is used to obtain the data quality difference degree between adjacent data nodes based on the data link quality assessment results. The link segmentation processing unit is used to segment the data link according to the data quality difference and generate data link segments of the same quality level. An anomaly identification unit is used to mark data link segments with a quality level lower than a preset quality threshold as anomalies, and to perform problem analysis on the marked quality anomaly link segments to obtain the anomaly feature identification parameters of the quality anomaly link segments. The processing parameter generation unit is used to identify parameters based on the abnormal features, analyze the impact range of the abnormal link segment and the processing measures for each node, and obtain the quality optimization processing parameters for the abnormal link segment.

Images