Data quality detection method and device, electronic equipment and computer readable medium

Abstract

The embodiment of the invention discloses a data quality detection method and device, electronic equipment and a computer readable medium. The method comprises the steps of generating a target network based on an association relationship between data nodes in a big data platform; determining a node centrality score of each data node in the target network based on the network structure of the target network; determining a first quality detection result of each data node in the target network based on a preset quality inspection rule and the attribute information of each data node in the target network; determining a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; and determining a data quality detection result of the big data platform based on the second quality detection result of each data node in the target network. According to the embodiment, the accuracy of data quality detection is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and specifically to a data quality detection method, device, electronic device, and computer-readable medium. Background Art

[0002] Big data platforms are network platforms that provide services through content sharing, resource sharing, channel co-construction, and data sharing. However, with the complexity of data systems and the rapid growth of data volume, data quality has become the key to whether big data platforms can operate efficiently and reliably. Therefore, it is very important to detect, analyze, and manage the overall data quality of big data platforms.

[0003] In the prior art, the data quality of a big data platform can be evaluated through metadata, data attributes, etc. However, the complex relationship between data nodes is an important reason for the low data quality. The above method does not consider the relationship and mutual influence between data nodes, so the detection of data quality is not accurate enough. Summary of the invention

[0004] The embodiments of the present application provide a data quality detection method, device, electronic device, and computer-readable medium to improve the accuracy of data quality detection.

[0005] In a first aspect, an embodiment of the present application provides a data quality detection method, the method comprising: generating a target network based on data nodes and association relationships between data nodes in a big data platform; determining a node centrality score of each data node in the target network based on a network structure of the target network; determining a first quality detection result of each data node in the target network based on preset quality inspection rules and attribute information of each data node in the target network; determining a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; and determining a data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

[0006] In a second aspect, an embodiment of the present application provides a data quality detection device, which includes: a generation unit, which is used to generate a target network based on data nodes in a big data platform and the association relationship between data nodes; a first determination unit, which is used to determine the node centrality score of each data node in the target network based on the network structure of the target network, and the node centrality score includes at least one of the following: degree centrality score, eigenvector centrality score, betweenness centrality score; a second determination unit, which is used to determine the first quality detection result of each data node in the target network based on preset quality inspection rules and attribute information of each data node in the target network; a third determination unit, which is used to determine the second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; and a fourth determination unit, which is used to determine the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any embodiment of the first aspect.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any embodiment of the first aspect.

[0009] The data quality detection method, device, electronic device and computer-readable medium provided in the embodiments of the present application first generate a target network based on the data nodes in the big data platform and the association relationship between the data nodes; then, based on the network structure of the target network, determine the node centrality score of each data node in the target network; then, based on the preset quality inspection rules and the attribute information of each data node in the target network, determine the first quality inspection result of each data node in the target network; then, based on the node centrality score of each data node in the target network and the first quality inspection result, determine the second quality inspection result of each data node in the above-mentioned target network; finally, based on the second quality inspection result of each data node in the target network, determine the data quality inspection result of the big data platform. Since the low quality of a certain data node will affect other data nodes through the association relationship between data nodes, which may cause low data quality of the entire big data platform, the embodiment of the present application introduces the network structure corresponding to the big data platform into the detection and analysis of the data quality of the big data platform. In the process of data quality detection, not only the attributes of the data node itself are considered, but also the association relationship and mutual influence between the data nodes are considered. Therefore, it is possible to integrate richer and more comprehensive information in the data detection process, and deepen the analysis of data quality from the data resource network level, thereby improving the accuracy of data quality detection of the big data platform. Through accurate data quality detection, key problems and anomalies in data nodes can be discovered, and the high-quality development of data processing links and data applications can be further guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0011] Figure 1 is a flow chart of an embodiment of the data quality detection method of the present application;

[0012] Figure 2 is a flow chart of an embodiment of the data quality detection method of the present application;

[0013] Figure 3 is a structural schematic diagram of an embodiment of a data quality detection device of the present application;

[0014] Figure 4 It is a schematic diagram of the structure of an electronic device used to implement an embodiment of the present application. DETAILED DESCRIPTION

[0015] All actions to obtain signals, information or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0016] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0017] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0018] The data quality detection method of the embodiment of the present application can be applied to the data quality detection scenario of the big data platform.

[0019] Big data platform is a network platform that provides services through content sharing, resource sharing, channel co-construction and data sharing. It is a set of infrastructure for handling scenarios such as massive data storage, computing and real-time computing of uninterrupted streaming data, including but not limited to Hadoop (a distributed system infrastructure), Spark (a multi-purpose cluster computing system), Storm (a large-scale distributed stream computing platform), Flink (an open source computing platform for distributed data stream processing and batch data processing), Flume (an open source log collection, aggregation and transmission system), Kafka (an open source stream processing platform), etc.

[0020] The construction of a big data platform can adopt either open source platforms or commercial solutions. It can be deployed on either private clouds or public clouds. By integrating various technical tools and services, the establishment of a big data platform meets the requirements for massive data collection, storage, processing and analysis. By better managing and utilizing data resources, the big data platform provides users with the most comprehensive and comprehensive view possible, helping users discover the correlation between things, infer development trends and results, and thus achieve greater value. Since the big data platform can obtain data more comprehensively and maximize the value of data, the emergence of the big data platform is of great significance to the data processing and application of modern society.

[0021] However, with the complexity of data systems and the rapid growth of data volume, data quality has become the key to whether big data platforms can operate efficiently and reliably. First, big data platforms often have a wide range of data sources, and the data formats, structures, and sources are different, which makes it difficult to integrate and analyze data, thus affecting data quality and data quality detection. Secondly, big data platforms involve very complex data processing processes. The amount of data that big data platforms need to process is very large, the data types are various, and the complexity of data processing and analysis is high, which increases the probability of data errors, making it difficult to accurately detect and improve data quality. At the same time, low-quality data will not only burden the processing of massive data, but also make the data analysis results unreliable. Therefore, it is very important to detect, analyze, and manage the overall data quality of big data platforms.

[0022] Traditional data quality detection methods establish quantitative indicator calculation parameters through metadata, data attributes, etc., so as to evaluate the data quality of the big data platform. However, these methods do not take into account the correlation and mutual influence between data in the data link, resulting in an isolated view of the data quality of each data, and the analysis of data quality remains on the intrinsic or extrinsic attributes of the data itself. However, in the process of data collection and processing, the intricate correlation between data is an important reason for the low data quality. Therefore, it is necessary to incorporate the correlation between data nodes in the big data platform into the detection and analysis of data quality. The embodiments of the present application can achieve this function.

[0023] Please refer to Figure 1 , which shows a process 100 of an embodiment of a data quality detection method according to the present application. The data quality detection method can be applied to various electronic devices with data processing functions. For example, the above electronic devices may include but are not limited to electronic devices such as servers. The execution subject of the data quality detection method may be a processor in the above electronic device.

[0024] The data quality detection method comprises the following steps:

[0025] Step 101: Generate a target network based on data nodes and association relationships between data nodes in a big data platform.

[0026] In this embodiment, the data nodes in the big data platform can be divided based on the granularity of {data source, database, data table}.

[0027] A data source is the source of data, a device or raw media that provides some required data. All information for establishing a database connection is stored in the data source. By providing the correct data source name, the corresponding database connection can be found. Data sources are usually used to connect to actual databases. A database can have multiple data source connections, and each data source must correspond to a specific database. The use of data sources simplifies the management of database connections, allowing users to access databases through data source names without having to directly deal with complex connection details.

[0028] A database is a computer software system that stores and manages data according to data structures. It is a warehouse for storing data. A data table is the basic component of a database. A database can contain multiple data tables.

[0029] A data table is a structured table used to store and manage data in a database. Each data table has a unique table name that is used to identify and distinguish different data sets. Each column in a table is called a field, which is used to store a specific type of data, such as text, numbers, dates, etc. Each row represents a record that records information about a specific entity or event. In addition, a data table can also define a primary key to uniquely identify each record in the table. The value of the primary key is unique in the table and can be a combination of one or more fields.

[0030] In this embodiment, a target network can be generated based on the data nodes and the association relationship between the data nodes in the big data platform. If there is an association relationship between two data nodes, it can be determined that there is an edge between the two data nodes. Conversely, if there is no association relationship between the two data nodes, it can be determined that there is no edge between the two data nodes. The target network can be a complex network (CN) used to characterize the data nodes and the association relationship between the data nodes in the big data platform. A complex network is a network constructed by abstracting a real system into data nodes and edges, and has a complex structure and characteristics.

[0031] In some optional implementations of this embodiment, the above step 101 may specifically include the following sub-steps:

[0032] Sub-step 1011, generating a data node set based on the data nodes in the big data platform.

[0033] Sub-step 1012, for every two data nodes in the above big data platform, if there is a reference relationship or dependency relationship between the above two data nodes in the data link, it is determined that the above two data nodes are associated, and it is determined that there is an edge between the above two data nodes. Among them, the reference relationship refers to the data caliber definition that the next data references another data in the data application scenario such as indicator construction. The dependency relationship refers to the fact that the generation of one data depends on another data in the data processing link.

[0034] Sub-step 1013: Generate an edge set based on the edges between the data nodes. The edge set is a collection of edges between the data nodes.

[0035] Sub-step 1014, based on the data node set and the edge set, generates an adjacency matrix for representing the network structure of the target network. The adjacency matrix (Adjacency Matrix) can also be called a graph matrix, which is a matrix used to represent the relationship between data nodes.

[0036] As an example, the number of data nodes is n, where n is a positive integer, and the data nodes can be represented as n1, n2, n3, ..., n n At this time, the adjacency matrix (which can be represented by G) can be an n×n matrix. For example, if data node n1 is associated with data node n2, the element G in the first row and second column of G can be 12 Set to 1. For another example, if n1 is not associated with n3 data nodes, then the element G in the first row and third column of G can be 13 Set to 0.

[0037] It should be noted that the values ​​in the adjacency matrix are not limited to 1 or 0. Other values ​​can be set according to the type or degree of association between data nodes. For example, if data node n1 is associated with data node n2, the element G in the first row and second column of G can be set to G based on the type or degree of association between the two. 12 Set to a number greater than 0 and less than or equal to 1.

[0038] It should also be noted that the size of the adjacency matrix can also be set to other values ​​and is not limited to n×n.

[0039] By constructing an adjacency matrix, the data nodes and their relationships in the big data platform can be clearly and intuitively represented, thereby clearly and intuitively reflecting the data network structure of the big data platform.

[0040] Step 102: Based on the network structure of the target network, determine the node centrality score of each data node in the target network.

[0041] In this embodiment, node centrality is an important concept in network analysis, which is used to measure the importance or influence of nodes in the network. It reflects the core degree and influence of the node in the target network by measuring the position and connection of the node in the network. Nodes with high node centrality play an important role in the network and may be key hubs for information transmission or centers for resource control.

[0042] In this embodiment, the node centrality score is a score obtained by taking the node centrality as an indicator. The node centrality score includes at least one of the following: degree centrality score, eigenvector centrality score, and betweenness centrality score.

[0043] The degree centrality score is the score obtained by using degree centrality (DC) as an indicator. Degree centrality is the most direct metric to characterize the centrality of a node in network analysis and can measure the role of a node in the network.

[0044] The eigenvector centrality score is the score obtained by using the eigenvector centrality (EC) as an indicator. The eigenvector centrality is used to measure the overall influence of a node in the network.

[0045] The betweenness centrality score is the score obtained by taking the betweenness centrality (BC) as an indicator. The betweenness centrality is used to measure the control ability of a node in information transmission.

[0046] In some optional implementations of this embodiment, the node centrality score may include a degree centrality score. Each data node in the target network may be taken as a target data node in turn, and the score of the target data node may be determined by the following steps:

[0047] The first step is to determine the degree of the target data node based on the network structure of the target network. The degree of the target data node is the number of edges directly connected to the target data node, which can be recorded as Ndgree. For data nodes in the network, the larger the degree of the data node, the more data processing links and data application processes it affects, and the more important its position in the network. Therefore, the degree of a data node can be used to measure its importance in the data network.

[0048] The second step is to determine the degree centrality score of the target data node based on the degree of the target data node and the total number of data nodes in the target network. Here, the total number of data nodes in the target network can be recorded as n, where n is a positive integer. For the target data node i, its degree centrality score DC i See the following formula:

[0049] DCi =Ndgree / (n-1)

[0050] Since degree centrality is the most direct metric to characterize the centrality of data nodes in network analysis and can measure the role of nodes in the network, by calculating the degree centrality score, we can take into account the correlation and mutual influence between data nodes, and integrate richer and more comprehensive information in the data quality detection process, thereby improving the accuracy of data quality detection.

[0051] In some optional implementations of this embodiment, the node centrality score includes a feature vector centrality score. The node centrality score of each data node in the target network can be determined by the following steps:

[0052] The first step is to obtain the adjacency matrix used to characterize the network structure of the target network and the degree centrality score of each data node in the target network. The adjacency matrix, also known as the graph matrix, is a matrix used to represent the relationship between data nodes.

[0053] The second step is to generate a first vector based on the degree centrality score. Here, the degree centrality scores of the data nodes in the target network can be arranged in sequence to obtain the first vector.

[0054] In the third step, the following steps are iteratively performed: multiplying the adjacency matrix by the first vector to obtain a second vector; when the second vector does not meet the target condition, replacing the first vector with the second vector.

[0055] Specifically, the above adjacency matrix is ​​recorded as A, the first vector used in a certain iteration is recorded as X1, and the second vector updated in a certain iteration is recorded as X2. If the product of A and X1 does not change the relative size of each value in X1, it can be determined that the second condition meets the target condition. That is, when A×X1=λX1, X2 at this time meets the target condition. λ is a proportional constant.

[0056] In the fourth step, when a second vector that meets the target condition is obtained, the iteration is stopped, and the second vector that meets the target condition is determined as a feature vector, where the feature vector includes the feature vector centrality score of each data node in the target network.

[0057] Specifically, the eigenvector centrality score EC of data node i can be expressed by the following formula: i :

[0058]

[0059] Among them, a ij is the value of the i-th row and j-th column in the adjacency matrix, which is used to indicate whether data node i and data node j have an association relationship. ijis the value of the i-th row and j-th column in X1 above. c is the proportional constant.

[0060] Since eigenvector centrality can measure the overall influence of a node in the network, by calculating the eigenvector centrality score, the correlation and mutual influence between data nodes can be taken into account, and richer and more comprehensive information can be integrated in the data quality detection process, thereby improving the accuracy of data quality detection.

[0061] In some optional implementations of this embodiment, the node centrality score includes a betweenness centrality score. Each data node in the target network is taken as a target data node in turn, and the betweenness centrality score of the target data node is determined by the following steps:

[0062] The first step is to determine the shortest path between every two data nodes in the target network and obtain the total number of shortest paths. Specifically, the shortest path algorithm such as Dijkstra algorithm and Floyd-Warshall algorithm can be used to calculate the shortest path from any data node s to any data node t in the target network and count the total number of shortest paths. The total number of shortest paths can be recorded as d st .

[0063] The second step is to count the number of target shortest paths that pass through the target data node in the determined shortest paths, which can be recorded as d st ().

[0064] The third step is to determine the ratio of the total number of shortest paths to the number of target shortest paths as the betweenness centrality score of the target data node. For the target data node i, its betweenness centrality score BC i See the following formula:

[0065]

[0066] Betweenness centrality reflects the control ability of a node in the network. The higher the betweenness centrality, the more data processing and application links it controls, and the stronger the control ability over the data processing and application of the big data platform. When a data node is of low quality, the data quality of the data link in which it is located cannot be guaranteed. By calculating the betweenness centrality of each data node, the correlation and mutual influence between data nodes can be taken into account, and richer and more comprehensive information can be integrated in the process of data quality detection, thereby improving the accuracy of data quality detection.

[0067] Step 103: Determine a first quality inspection result of each data node in the target network based on a preset quality inspection rule and attribute information of each data node in the target network.

[0068] In this embodiment, each data node in the target network may first be quality inspected based on a preset quality inspection rule to obtain a quality inspection rule score. Then, based on the attribute information of each data node in the target network, an attribute score of each data node may be obtained. Finally, for each data node, a first quality inspection result of each data node in the target network may be determined based on the quality inspection rule score and the attribute score of the data node.

[0069] Specifically, please refer to the following sub-steps:

[0070] Sub-step 1031 , based on the preset quality inspection rules, quality inspection is performed on each data node in the target network to obtain a quality inspection rule score.

[0071] Here, data quality inspection rules can be pre-configured for the big data platform as needed. The above quality inspection rules can be expanded from the dimensions of timeliness, uniqueness, consistency, effectiveness, completeness and accuracy. For example, the trend of data changes can be detected by comparing statistical data in different time periods. The quality inspection results of each data node can be obtained by periodically executing or manually executing the quality inspection rules through the big data platform.

[0072] One or more quality inspection rules can be set. The quality inspection result corresponding to a single quality inspection rule j of data node i is R ij , when the quality inspection passes, R ij =1; when the quality inspection fails, R ij = 0. If the total number of quality inspection rules is N, and N is a positive integer, then based on the preset quality inspection rules, the quality inspection rule score R of data node i can be obtained. i , see the following formula:

[0073]

[0074] Sub-step 1032, based on the attribute information of each data node in the target network, determine the attribute score of each data node in the target network. Here, the attribute information may include but is not limited to data scale, data asset level, etc. The attribute information may be normalized to obtain the attribute score D corresponding to data node i i .

[0075] Sub-step 1033, for each data node in the target network, based on the quality inspection rule score and attribute score of the data node, determine the first quality inspection result of the data node. Specifically, for data node i, its first quality inspection result can be the above-mentioned quality inspection rule score R i and the average of the attribute scores D i , that is: (R i +D i ) / 2.

[0076] Step 104: Determine a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result.

[0077] In this embodiment, for each data node in the target network, the node centrality score of the data node may be used as a weight, and the first quality detection result of the data node may be weighted based on the weight to obtain the second quality detection result of the data node.

[0078] Specifically, the node centrality scores of each data node in the target network include degree centrality score, eigenvector centrality score, and betweenness centrality score. For data node i, its degree centrality score, eigenvector centrality score, and betweenness centrality score are respectively denoted as DC i ,EC i , B.C. i . You can first use DC i ,EC i , B.C. i After normalization, the average is calculated to obtain the data quality weight CN of each data node in the target network. i The weight represents the impact of the data quality corresponding to the data node i on the overall data quality of the target network. It can be used to characterize the importance of the data node i in the target network and to distinguish whether the data node i is an important data node in the target network. When the data quality of an important data node decreases, it affects the production and application of other data nodes through the target network, which may seriously reduce the overall data quality of the big data platform.

[0079] After getting the data quality weight CN i After that, based on the weight CN i The first quality detection result (R i +D i ) / 2 for weighted calculation to obtain the second quality detection result of data node i. The second quality detection result of data node i can be recorded as G i , then:

[0080] G i =CN i ×(R i +D i ) / 2

[0081] Step 105: Determine the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

[0082] In this embodiment, a preset rule may be used to merge the second quality detection results of each data node in the target network to determine the data quality detection result of the big data platform.

[0083] In some optional implementations of this embodiment, the second quality detection results of each data node may be weighted and summed to obtain the data quality detection result of the big data platform. The data quality detection result may be denoted as G, and then:

[0084]

[0085] By testing the data quality of the big data platform, the data quality of the big data platform can be tested and evaluated as a whole from the network level, so as to facilitate the updating and improvement of the big data platform.

[0086] The method provided by the above-mentioned embodiment of the present application first generates a target network based on the data nodes in the big data platform and the association relationship between the data nodes; then, based on the network structure of the target network, determines the node centrality score of each data node in the target network; thereafter, based on the preset quality inspection rules and the attribute information of each data node in the target network, determines the first quality inspection result of each data node in the target network; then, based on the node centrality score of each data node in the target network and the first quality inspection result, determines the second quality inspection result of each data node in the above-mentioned target network; finally, based on the second quality inspection result of each data node in the target network, determines the data quality inspection result of the big data platform. Since the low quality of a certain data node will affect other data nodes through the association relationship between data nodes, which may cause low data quality of the entire big data platform, the embodiment of the present application introduces the network structure corresponding to the big data platform into the detection and analysis of the data quality of the big data platform. In the process of data quality detection, not only the attributes of the data node itself are considered, but also the association relationship and mutual influence between the data nodes are considered. Therefore, it is possible to integrate richer and more comprehensive information in the data detection process, and deepen the analysis of data quality from the data resource network level, thereby improving the accuracy of data quality detection of the big data platform. Through accurate data quality detection, key problems and anomalies in data nodes can be discovered, and the high-quality development of data processing links and data applications can be further guaranteed.

[0087] Further references Figure 2 , which shows a process 200 of another embodiment of a data quality detection method. The process 200 of the data quality detection method includes the following steps:

[0088] Step 201: Generate a target network based on data nodes and association relationships between data nodes in a big data platform.

[0089] Step 201 of this embodiment can be found in Figure 1 The step 101 of the corresponding embodiment will not be described in detail here.

[0090] In some optional implementations of the present embodiment, the target network is generated based on the data nodes in the big data platform and the association relationship between the data nodes, including: generating a data node set based on the data nodes in the big data platform; for every two data nodes in the big data platform, if there is a reference relationship or a dependency relationship between the two data nodes in the data link, it is determined that there is an association relationship between the two data nodes, and it is determined that there is an edge between the two data nodes; based on the edges between the data nodes, an edge set is generated; based on the data node set and the edge set, an adjacency matrix for characterizing the network structure of the target network is generated.

[0091] Step 202: Based on the network structure of the target network, determine the node centrality score of each data node in the target network.

[0092] Step 202 of this embodiment can be found in Figure 1 The step 102 of the corresponding embodiment will not be described in detail here.

[0093] In some optional implementations of the present embodiment, the node centrality score includes a degree centrality score; the node centrality score of each data node in the target network based on the network structure of the target network includes: taking each data node in the target network as a target data node in turn, and executing the following steps: determining the degree of the target data node based on the network structure of the target network; determining the degree centrality score of the target data node based on the degree of the target data node and the total number of data nodes in the target network.

[0094] In some optional implementations of this embodiment, the node centrality score includes a eigenvector centrality score; the node centrality score of each data node in the target network is determined based on the network structure of the target network, including: obtaining an adjacency matrix for characterizing the network structure of the target network, and the degree centrality score of each data node in the target network; generating a first vector based on the degree centrality score; iteratively performing the following steps: multiplying the adjacency matrix by the first vector to obtain a second vector; when the second vector does not meet the target condition, replacing the first vector with the second vector. When the second vector that meets the target condition is obtained, the iteration is stopped, and the second vector that meets the target condition is determined as a eigenvector, and the eigenvector includes the eigenvector centrality score of each data node in the target network.

[0095] In some optional implementations of the present embodiment, the node centrality score includes an betweenness centrality score; the node centrality score of each data node in the target network is determined based on the network structure of the target network, including: taking each data node in the target network as a target data node in turn, and executing the following steps: determining the shortest path between every two data nodes in the target network to obtain the total number of shortest paths; counting the number of target shortest paths passing through the target data node in the determined shortest paths; and determining the ratio of the total number of shortest paths to the number of target shortest paths as the betweenness centrality score of the target data node.

[0096] Step 203: Determine a first quality inspection result of each data node in the target network based on a preset quality inspection rule and attribute information of each data node in the target network.

[0097] Step 203 of this embodiment can be found in Figure 1 The step 103 of the corresponding embodiment will not be described in detail here.

[0098] Step 204: Determine a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result.

[0099] Step 204 of this embodiment can be found in Figure 1 The step 104 of the corresponding embodiment will not be described in detail here.

[0100] Step 205: Determine the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

[0101] Step 205 of this embodiment can be found in Figure 1 The step 105 of the corresponding embodiment will not be described in detail here.

[0102] In some optional implementations of this embodiment, the following steps may be performed: weighted summing of the second quality detection results of each data node is performed to obtain a data quality detection result of the big data platform.

[0103] Step 206: sort the data nodes in the target network based on the second quality detection result of each data node in the target network to obtain a sorting result.

[0104] In this embodiment, since the second quality detection result is a numerical value, the higher the numerical value, the greater the impact on the data quality. Therefore, the data nodes can be sorted in descending order of the second quality detection result to obtain a sorting result.

[0105] Step 207: Based on the sorting result, determine the key data nodes in the network structure and mark the key data nodes.

[0106] In this embodiment, a specified number of data nodes ranked high can be selected as key data nodes, or data nodes whose second quality detection results are greater than a preset threshold can be selected as key data nodes.

[0107] Since the low data quality of key data nodes not only affects the data node itself, but also reduces the processing and application quality of other data nodes through the data link, thus causing the data quality of the entire big data platform to be low. Therefore, marking key data nodes can facilitate timely evaluation and improvement of key data nodes.

[0108] from Figure 2 It can be seen that Figure 1 Compared with the corresponding embodiment, the process 200 of the data quality detection method in this embodiment involves the steps of determining and marking the key data nodes in the big data platform. Therefore, the scheme described in this embodiment can facilitate timely evaluation and improvement of the data of the key data nodes.

[0109] Further references Figure 3 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a data quality detection device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0110] like Figure 3 As shown, the data quality detection device 300 of this embodiment includes: a generation unit 301, which is used to generate a target network based on data nodes in a big data platform and the association relationship between data nodes; a first determination unit 302, which is used to determine the node centrality score of each data node in the target network based on the network structure of the target network; a second determination unit 303, which is used to determine the first quality detection result of each data node in the target network based on a preset quality inspection rule and the attribute information of each data node in the target network; a third determination unit 304, which is used to determine the second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; a fourth determination unit 305, which is used to determine the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

[0111] In some optional implementations of this embodiment, the generation unit 301 is also used to: generate a data node set based on the data nodes in the big data platform; for every two data nodes in the big data platform, if there is a reference relationship or a dependency relationship between the two data nodes in the data link, determine that the two data nodes have an association relationship, and determine that there is an edge between the two data nodes; generate an edge set based on the edges between the data nodes; based on the data node set and the edge set, generate an adjacency matrix for characterizing the network structure of the target network.

[0112] In some optional implementations of this embodiment, the node centrality score includes a degree centrality score; the first determination unit 302 is also used to: take each data node in the target network as a target data node in turn, and perform the following steps: determine the degree of the target data node based on the network structure of the target network; determine the degree centrality score of the target data node based on the degree of the target data node and the total number of data nodes in the target network.

[0113] In some optional implementations of this embodiment, the node centrality score includes a eigenvector centrality score; the first determination unit 302 is further used to: obtain an adjacency matrix for characterizing the network structure of the target network, and the degree centrality score of each data node in the target network; generate a first vector based on the degree centrality score; iteratively perform the following steps: multiply the adjacency matrix by the first vector to obtain a second vector; when the second vector does not meet the target condition, replace the first vector with the second vector. When the second vector that meets the target condition is obtained, stop the iteration, determine the second vector that meets the target condition as a eigenvector, and the eigenvector includes the eigenvector centrality score of each data node in the target network.

[0114] In some optional implementations of this embodiment, the node centrality score includes the betweenness centrality score; the first determination unit 302 is also used to: take each data node in the target network as the target data node in turn, and perform the following steps: determine the shortest path between every two data nodes in the target network to obtain the total number of shortest paths; count the number of target shortest paths passing through the target data node in the determined shortest paths; and determine the ratio of the total number of shortest paths to the number of target shortest paths as the betweenness centrality score of the target data node.

[0115] In some optional implementations of this embodiment, the fourth determination unit 305 is further used to perform weighted summation on the second quality detection results of each data node in the target network to obtain the data quality detection result of the big data platform.

[0116] In some optional implementations of this embodiment, the device also includes: a fifth determination unit, used to sort each data node based on the second quality detection result of each data node to obtain a sorting result; based on the sorting result, determine the key data nodes in the network structure, and mark the key data nodes.

[0117] The device provided by the above-mentioned embodiment of the present application first generates a target network based on the data nodes in the big data platform and the association relationship between the data nodes; then, based on the network structure of the target network, determines the node centrality score of each data node in the target network; thereafter, based on the preset quality inspection rules and the attribute information of each data node in the target network, determines the first quality inspection result of each data node in the target network; then, based on the node centrality score of each data node in the target network and the first quality inspection result, determines the second quality inspection result of each data node in the above-mentioned target network; finally, based on the second quality inspection result of each data node in the target network, determines the data quality inspection result of the big data platform. Since the low quality of a certain data node will affect other data nodes through the association relationship between data nodes, which may cause low data quality of the entire big data platform, the embodiment of the present application introduces the network structure corresponding to the big data platform into the detection and analysis of the data quality of the big data platform. In the process of data quality detection, not only the attributes of the data node itself are considered, but also the association relationship and mutual influence between the data nodes are considered. Therefore, it is possible to integrate richer and more comprehensive information in the data detection process, and deepen the analysis of data quality from the data resource network level, thereby improving the accuracy of data quality detection of the big data platform. Through accurate data quality detection, key problems and anomalies in data nodes can be discovered, and the high-quality development of data processing links and data applications can be further guaranteed.

[0118] Reference below Figure 4 , which shows a schematic structural diagram of an electronic device for implementing some embodiments of the present application. Figure 4 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Figure 4As shown, the electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0120] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a disk, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0121] In particular, according to some embodiments of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of some embodiments of the present application are executed.

[0122] It should be noted that the computer-readable medium described in some embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0123] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (Hyper Text Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0124] The computer-readable medium may be included in the electronic device; or it may exist independently without being assembled into the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: generates a target network based on the data nodes and the association relationship between the data nodes in the big data platform; determines the node centrality score of each data node in the target network based on the network structure of the target network, and the node centrality score includes at least one of the following: degree centrality score, eigenvector centrality score, and betweenness centrality score; determines the first quality inspection result of each data node in the target network based on the preset quality inspection rules and the attribute information of each data node in the target network; determines the second quality inspection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality inspection result.

[0125] Computer program code for performing the operations of some embodiments of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++; and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, or may be connected to an external computer (e.g., through the Internet using an Internet service provider), including a local area network (LAN) or a wide area network (WAN).

[0126] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0127] The units described in some embodiments of the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a first determination unit, a second determination unit, a selection unit, and a third determination unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0128] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0129] The above description is only some preferred embodiments of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present application is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with (but not limited to) the technical features with similar functions disclosed in the embodiments of the present application to form a technical solution.

Claims

1. A data quality detection method, characterized in that: The method comprises: Generate a target network based on the data nodes and the association relationship between the data nodes in the big data platform; Based on the network structure of the target network, determining the node centrality score of each data node in the target network; Determine a first quality inspection result of each data node in the target network based on a preset quality inspection rule and attribute information of each data node in the target network; Determining a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; Based on the second quality detection result of each data node in the target network, the data quality detection result of the big data platform is determined.

2. The method according to claim 1, characterized in that The generating of the target network based on the data nodes and the association relationship between the data nodes in the big data platform includes: Generate a data node set based on the data nodes in the big data platform; For every two data nodes in the big data platform, if there is a reference relationship or a dependency relationship between the two data nodes in the data link, it is determined that there is an association relationship between the two data nodes, and it is determined that there is an edge between the two data nodes; Generate edge sets based on the edges between data nodes; Based on the data node set and the edge set, an adjacency matrix for representing the network structure of the target network is generated.

3. The method according to claim 1, characterized in that The node centrality score includes a degree centrality score; and determining the node centrality score of each data node in the target network based on the network structure of the target network includes: Take each data node in the target network as the target data node in turn and perform the following steps: Determining the degree of the target data node based on the network structure of the target network; A degree centrality score of the target data node is determined based on the degree of the target data node and the total number of data nodes of the target network.

4. The method according to claim 1, characterized in that: The node centrality score includes a feature vector centrality score; and determining the node centrality score of each data node in the target network based on the network structure of the target network includes: Obtaining an adjacency matrix for characterizing the network structure of the target network and a degree centrality score of each data node in the target network; Based on the degree centrality score, generating a first vector; Iteratively performing the following steps: multiplying the adjacency matrix by the first vector to obtain a second vector; when the second vector does not meet the target condition, replacing the first vector with the second vector; When a second vector that meets the target condition is obtained, the iteration is stopped, and the second vector that meets the target condition is determined as a feature vector, where the feature vector includes a feature vector centrality score of each data node in the target network.

5. The method according to claim 1, characterized in that The node centrality score includes a betweenness centrality score; and determining the node centrality score of each data node in the target network based on the network structure of the target network includes: Take each data node in the target network as the target data node in turn and perform the following steps: Determine the shortest path between every two data nodes in the target network to obtain the total number of shortest paths; Counting the number of target shortest paths passing through the target data node in the determined shortest paths; The ratio of the total number of shortest paths to the number of target shortest paths is determined as the betweenness centrality score of the target data node.

6. The method according to claim 1, characterized in that The determining the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network includes: A weighted sum is performed on the second quality detection result of each data node in the target network to obtain a data quality detection result of the big data platform.

7. The method according to claim 1, characterized in that After determining the second quality detection result of each data node in the target network, the method further includes: Based on the second quality detection result of each data node in the target network, sorting each data node in the target network to obtain a sorting result; Based on the sorting result, key data nodes in the network structure are determined, and the key data nodes are marked.

8. A data quality detection device, characterized in that: The device comprises: A generating unit, used for generating a target network based on data nodes and association relationships between data nodes in the big data platform; A first determining unit, configured to determine a node centrality score of each data node in the target network based on the network structure of the target network; A second determining unit, configured to determine a first quality detection result of each data node in the target network based on a preset quality inspection rule and attribute information of each data node in the target network; A third determining unit, configured to determine a second quality detection result of each data node in the target network based on the node centrality score of each data node in the target network and the first quality detection result; The fourth determination unit is used to determine the data quality detection result of the big data platform based on the second quality detection result of each data node in the target network.

9. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images