Data cleaning method, device and equipment based on graph algorithm and storage medium

Abstract

The invention relates to a data cleaning method and device based on a graph algorithm, equipment and a storage medium, and relates to the field of data processing. The method comprises the steps of receiving to-be-cleaned target data, and collecting blood relationship information of the target data; generating an association graph network of the target data according to the blood relationship information; searching source-attached layer data of the target data by using a graph algorithm, and formulating a cleaning rule of the source-attached layer data; storing the plurality of cleaning rules in a preset cleaning rule base; and cleaning the target data according to the cleaning rule base and the associated atlas network. The technical effect of the invention is that the data cleaning efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data cleaning method, device, equipment and storage medium based on a graph algorithm. Background Art

[0002] Data cleaning refers to a series of operations to check, correct and standardize raw data during data analysis and processing to ensure the accuracy, consistency and integrity of the data. In the data cleaning method based on graph algorithms, the definition of graph algorithms can be concretized as a technical method that uses graph data structures and related algorithms to identify, correct and optimize errors, missing values ​​and outliers in data sets.

[0003] Existing data cleaning methods are usually based on principles such as rules, pattern matching, and anomaly detection. The main steps include data quality assessment, outlier detection and processing, missing value filling, duplicate data processing, etc. These methods work well when processing simple data and clear rules, but are often inefficient when facing complex data and data sets with complex relationships. Summary of the invention

[0004] In order to improve data cleaning efficiency, the present application provides a data cleaning method, device, equipment and storage medium based on a graph algorithm.

[0005] In the first aspect, the present application provides a data cleaning method based on a graph algorithm, which adopts the following technical solution:

[0006] receiving target data to be cleaned, and collecting blood relationship information of the target data;

[0007] Generate a correlation graph network of the target data according to the blood relationship information;

[0008] Using a graph algorithm to find the source layer data of the target data, and formulating cleaning rules for the source layer data;

[0009] Storing the plurality of cleaning rules in a preset cleaning rule library;

[0010] The target data is cleaned according to the cleaning rule base and the association graph network.

[0011] Through the above technical solution, cleaning rules for source layer data are formulated and combined with the associated graph network for data cleaning. When faced with complex data and data sets with complex relationships, data can be cleaned quickly to improve data cleaning efficiency.

[0012] In a specific implementation scheme, the graph algorithm includes a graph traversal algorithm and a conditional screening algorithm, and the use of the graph algorithm to search for the source layer data of the target data includes:

[0013] Analyzing the association graph network using the graph traversal algorithm to obtain an analysis result;

[0014] Formulate screening conditions for the source layer data according to the analysis results;

[0015] The source layer data of the target data is obtained by filtering according to the filtering conditions using a conditional filtering algorithm.

[0016] Through the above technical solution, a graph algorithm is used to analyze the associated graph network, and the data segments in which the relationships are clustered are found to be the source layer data, thereby improving the accuracy of the source layer data and improving the cleaning efficiency.

[0017] In a specific implementation scheme, the formulating of cleaning rules for the source layer data includes:

[0018] Determine the rule configuration content according to the source layer data, wherein the rule configuration content includes the rule type and the rule content;

[0019] A cleaning rule for the source layer data is formulated according to the rule configuration content.

[0020] Through the above technical solution, the cleaning rule configuration content is first determined, and the rule type and rule content are clarified, so that the cleaning rules are clear and the accuracy of data cleaning is improved.

[0021] In a specific implementation scheme, cleaning the target data according to the cleaning rule base and the association graph network includes:

[0022] Using a graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network;

[0023] formulating a downstream associated field cleaning rule for the downstream associated field using the cleaning rule;

[0024] Adding the downstream associated field cleaning rule to the cleaning rule base to obtain a target cleaning rule base;

[0025] The target data is cleansed according to the target cleaning rule base to generate cleansed data.

[0026] Through the above technical solution, cleaning rules for the source layer data are formulated and a data cleaning rule library is pre-established in combination with the associated graph network, thereby improving data cleaning efficiency.

[0027] In a specific implementation scheme, the graph algorithm includes a depth-first search algorithm, and the use of the graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network includes:

[0028] Analyzing the association graph network using the depth-first search algorithm to obtain the data association relationship of the target data;

[0029] The downstream associated field of the source layer data is found according to the data association relationship.

[0030] Through the above technical solution, the depth-first search algorithm is used to find the downstream associated fields of the source layer data, making the search result more accurate.

[0031] In a specific implementation scheme, the target data includes a plurality of target data segments, and cleaning the target data according to the target cleaning rule base includes:

[0032] When a target data segment to be cleaned is received, searching the target cleaning rule for the target data segment in the target cleaning rule library;

[0033] Determining a cleaning method for the target data segment according to the cleaning rule;

[0034] Cleaning the target data segment using the cleaning method according to the cleaning rule;

[0035] When the cleaning of the target data segments is completed, cleaned data is generated.

[0036] Through the above technical solution, a cleaning method is selected according to the cleaning rules, and a script or data cleaning tool is used to clean the data, thereby improving the data cleaning efficiency.

[0037] In a specific implementation scheme, after cleaning the target data according to the target cleaning rule base to generate cleansed data, the method further includes:

[0038] Using a data quality check tool to verify the cleaned data to obtain a verification result, and determining whether the verification result is passed;

[0039] If the verification result is passed, the data cleaning is successful;

[0040] Otherwise, data cleaning fails and data cleaning is performed again.

[0041] Through the above technical solution, the cleaned data is verified to ensure the accuracy and completeness of data cleaning.

[0042] In a second aspect, the present application provides a data cleaning device based on a graph algorithm, which adopts the following technical solution: the device comprises:

[0043] A blood relationship information collection module, used to receive target data to be cleaned and collect blood relationship information of the target data;

[0044] An association graph network generation module, used to generate an association graph network of the target data according to the blood relationship information;

[0045] A source layer data processing module, used for searching the source layer data of the target data by using a graph algorithm and formulating cleaning rules for the source layer data;

[0046] A cleaning rule base generation module, used for storing a number of the cleaning rules into a preset cleaning rule base;

[0047] A data cleaning module is used to clean the target data according to the cleaning rule base and the associated graph network.

[0048] In a third aspect, the present application provides a computer device, which adopts the following technical solution: it includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executes a data cleaning method based on a graph algorithm as described above.

[0049] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: storing a computer program that can be loaded by a processor and execute the above-mentioned graph algorithm-based data cleaning method.

[0050] In summary, this application has the following beneficial technical effects:

[0051] (1) Through blood relationship and graph algorithm, the relationship between source layer data and downstream data is identified, and cleaning rules are automatically generated to realize automatic cleaning of data, reducing manual intervention.

[0052] (2) By using graph algorithms to analyze and mine data relationships, we can quickly generate cleaning rules for subsequent processing links based on the source layer cleaning rules. When faced with complex data and data sets with complex relationships, we can also quickly clean data and improve data cleaning efficiency.

[0053] (3) The method of this application does not depend on a specific data type or field and is applicable to data sets of various types and sizes, with strong versatility and applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flow chart of a data cleaning method based on a graph algorithm in an embodiment of the present application.

[0055] Figure 2 A structural block diagram of a data cleaning method based on a graph algorithm in an embodiment of the present application.

[0056] Figure numerals: 201, blood relationship information collection module; 202, association graph network generation module; 203, source layer data processing module; 204, cleaning rule base generation module; 205, data cleaning module. DETAILED DESCRIPTION

[0057] The following is combined with Figure 1-Figure 2 This application is described in further detail.

[0058] The embodiments of the present application disclose a data cleaning method, device, equipment and storage medium based on a graph algorithm, which is used to improve data cleaning efficiency.

[0059] Data cleaning refers to a series of operations to check, correct and standardize raw data during data analysis and processing to ensure the accuracy, consistency and integrity of the data. In the data cleaning method based on graph algorithms, the definition of graph algorithms can be concretized as a technical method that uses graph data structures and related algorithms to identify, correct and optimize errors, missing values ​​and outliers in data sets.

[0060] Existing data cleaning methods are usually based on principles such as rules, pattern matching, and anomaly detection. The main steps include data quality assessment, outlier detection and processing, missing value filling, duplicate data processing, etc. These methods work well when processing simple data and clear rules, but are often inefficient when facing complex data and data sets with complex relationships.

[0061] Therefore, this application proposes a data cleaning method based on graph algorithm, which is used to improve data cleaning efficiency.

[0062] like Figure 1 As shown, the method includes:

[0063] S10, receiving target data to be cleaned, and collecting blood relationship information of the target data.

[0064] Specifically, when receiving the target data that needs to be cleaned, it is necessary to first collect the lineage information of the target data source. The lineage here refers to the source and dependency relationship between the data, including data tables, fields, associations, etc.

[0065] S20, generating a correlation graph network of the target data according to the blood relationship information.

[0066] Specifically, based on the collected blood relationship information, the structure of the association graph network is defined, which includes determining the types and attributes of nodes (representing data items or data sources) and edges (representing the dependencies between data). Through this association graph network, the data flow relationship between fields and the main data processing logic can be clearly reflected. This can help us better understand the relationship between data, thereby providing a basis for more accurate generation of downstream association field cleaning rules.

[0067] S30, using a graph algorithm to find the source layer data of the target data, and formulating cleaning rules for the source layer data.

[0068] Specifically, the source layer data is the starting point of data processing, providing a foundation for subsequent data analysis, mining, and business applications. The graph algorithm is used to analyze the associated graph network to find the source layer data of the target data, and then a number of corresponding cleaning rules are formulated for different source layer data.

[0069] S40, storing a plurality of cleaning rules in a preset cleaning rule library.

[0070] Specifically, a cleaning rule library is established in advance, and a number of cleaning rules corresponding to different source layer data are stored in the cleaning rule library, so as to facilitate calling the cleaning rules during data cleaning.

[0071] S50, cleaning the target data according to the cleaning rule base and the associated graph network.

[0072] Specifically, when cleaning the target data, first find the downstream associated fields corresponding to the different source layer data according to the associated graph network. Generally speaking, the cleaning rules of the downstream associated fields are consistent with the cleaning rules of the source layer data, and call the cleaning rule library to clean the target data.

[0073] This application develops cleaning rules for source layer data and combines it with an associated graph network for data cleaning. It can also quickly clean data when faced with complex data and data sets with complex relationships, thereby improving data cleaning efficiency.

[0074] In one embodiment, in order to improve data cleaning efficiency, the graph algorithm includes a graph traversal algorithm and a conditional screening algorithm. The step of using the graph algorithm to find the source layer data of the target data can be specifically performed as follows:

[0075] First, the association graph network is analyzed using a graph traversal algorithm to obtain the analysis results. Specifically, a graph traversal algorithm is a type of graph algorithm that is used to systematically access all nodes in the association graph network. The association graph network includes determining the types and properties of nodes (representing data items or data sources) and edges (representing dependencies between data).

[0076] Then, according to the analysis results, the filtering conditions of the source layer data are formulated. Specifically, the source layer data is the starting point of data processing, so the source layer data is represented as only downstream data in the associated graph network. The filtering conditions are determined to filter the target data.

[0077] Next, a conditional screening algorithm is used to filter the source layer data of the target data according to the screening conditions. Specifically, the data with only downstream data found in the associated graph network is the source layer data of the target data.

[0078] By analyzing the associated graph network through graph algorithms, the data segments in which the relationships are clustered are found to be the source layer data, thereby improving the accuracy of the source layer data and improving the cleaning efficiency.

[0079] In one embodiment, in order to improve data cleaning efficiency, the step of formulating cleaning rules for source layer data can be specifically performed as follows:

[0080] First, the rule configuration content is determined according to the source layer data. The rule configuration content includes the rule type and the rule content. Specifically, several source layer data are analyzed to clarify the rule configuration content. The rule types include supplementing missing values, deleting duplicates, processing outliers, code value conversion, normalization, etc. The rule content specifically points out how to modify it.

[0081] Then, a cleaning rule for the source layer data is formulated according to the rule configuration content. Specifically, a number of cleaning rules corresponding to the source layer data are formulated according to the rule type and rule content.

[0082] This application first determines the cleaning rule configuration content, clarifies the rule type and rule content, makes the cleaning rules clear, and improves the accuracy of data cleaning.

[0083] In one embodiment, in order to improve data cleaning efficiency, the step of cleaning target data according to the cleaning rule base and the associated graph network can be specifically performed as follows:

[0084] First, we use graph algorithms to find the downstream associated fields of the source layer data based on the associated graph network. Specifically, the associated graph network includes determining the types and attributes of nodes (representing data items or data sources) and edges (representing the dependencies between data). We use graph algorithms to find the branches corresponding to each source layer data, i.e., the downstream associated fields.

[0085] Then, the cleaning rules are used to formulate the downstream associated field cleaning rules of the downstream associated fields. Specifically, the cleaning rules of the downstream associated fields are consistent with the cleaning rules of the source layer data. The cleaning rules of the source layer data are applied to the downstream associated fields to obtain the downstream associated field cleaning rules.

[0086] Next, the downstream associated field cleaning rules are added to the cleaning rule library to obtain the target cleaning rule library, and the target data is cleaned according to the target cleaning rule library to generate cleaned data. Specifically, the downstream associated field cleaning rules are added to the preset cleaning rule library, and the cleaning rule library originally also contains the cleaning rules for the source layer data, thus forming a comprehensive cleaning rule library for the target data, and the target data is cleaned according to the target cleaning rule library to generate cleaned data.

[0087] The present application formulates cleaning rules for source layer data and pre-establishes a data cleaning rule base in conjunction with an associated graph network, thereby improving data cleaning efficiency.

[0088] In one embodiment, in order to improve the efficiency of data cleaning, the graph algorithm includes a depth-first search algorithm. The step of using the graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network can be specifically performed as follows:

[0089] The depth-first search algorithm is used to analyze the association graph network to obtain the data association relationship of the target data, and the downstream association fields of the source layer data are found according to the data association relationship. Specifically, the association graph network includes determining the types and attributes of nodes (representing data items or data sources) and edges (representing the dependencies between data). Depth-First Search (DFS) is an algorithm for traversing or searching trees or graphs. It searches as deep as possible along each branch of the graph until it reaches a leaf node or the end of the graph, then backtracks and continues to search other unvisited branches. During the DFS traversal process, the association relationship between access paths or nodes, that is, the data association relationship, can be recorded. Through the data association relationship and the source layer data, the downstream association fields of the source layer data can be obtained.

[0090] This application uses a depth-first search algorithm to find downstream associated fields of source layer data to make the search results more accurate.

[0091] In one embodiment, in order to improve data cleaning efficiency, the target data includes several target data segments, and the step of cleaning the target data according to the target cleaning rule library can be specifically performed as follows:

[0092] First, when a target data segment to be cleaned is received, a target cleaning rule of the target data segment is searched in a target cleaning rule library. Specifically, several target data segments are involved in the data cleaning process. After the cleaning rule library is established, the cleaning rule of the target data segment will be included in the cleaning rule library.

[0093] Next, determine the cleaning method of the target data segment based on the cleaning rules. Specifically, the cleaning rules of the target data segment will be included in the cleaning rule library. Different cleaning rules have different cleaning methods. The cleaning methods mainly include script cleaning or automated processing using data cleaning tools. Therefore, before cleaning, it is necessary to determine the cleaning method of the target data segment based on the cleaning rules.

[0094] Next, the target data segments are cleaned using a cleaning method according to the cleaning rules, and cleaned data is generated after cleaning of several target data segments. Specifically, the cleaned data is obtained after all the target data segments are automatically cleaned using a cleaning method according to the cleaning rules.

[0095] This application selects a cleaning method according to the cleaning rules and uses scripts or data cleaning tools to clean the data, thereby improving the efficiency of data cleaning.

[0096] In one embodiment, in order to improve data cleaning efficiency, after cleaning the target data according to the target cleaning rule library to generate cleaned data, the following steps may be performed:

[0097] Use data quality checking tools to verify the cleaned data to obtain verification results and determine whether the verification results are passed. Specifically, verify whether all target data are cleaned according to the cleaning rules and achieve the cleaning purpose. You can use data quality checking tools such as Data Quality Checker, Talend Data Quality, etc. to verify the data and obtain verification results.

[0098] If the verification result is passed, the data cleaning is successful and the data cleaning is completed.

[0099] On the contrary, data cleaning fails and data cleaning needs to be repeated. The problem can be fed back to the data cleaning team for corresponding adjustments and optimizations, and then data verification can be performed again until the data reaches the expected quality standards.

[0100] This application verifies the cleaned data to ensure the accuracy and completeness of the data cleansing.

[0101] Based on the above method, the present application also discloses a data cleaning device based on a graph algorithm. Figure 2 , the device includes the following modules:

[0102] The blood relationship information collection module 201 is used to receive the target data to be cleaned and collect the blood relationship information of the target data;

[0103] The association graph network generation module 202 is used to generate an association graph network of target data according to blood relationship information;

[0104] The source layer data processing module 203 is used to use a graph algorithm to find the source layer data of the target data and formulate a cleaning rule for the source layer data;

[0105] A cleaning rule base generation module 204 is used to store a number of cleaning rules into a preset cleaning rule base;

[0106] The data cleaning module 205 is used to clean the target data according to the cleaning rule base and the associated graph network.

[0107] In one embodiment, the source layer data processing module 203 is specifically used to use a graph traversal algorithm to analyze the associated graph network to obtain analysis results; formulate filtering conditions for the source layer data based on the analysis results; and use a conditional filtering algorithm to filter and obtain the source layer data of the target data based on the filtering conditions.

[0108] In one embodiment, the source layer data processing module 203 is specifically used to determine the rule configuration content according to the source layer data, the rule configuration content includes the rule type and the rule content; and formulate the source layer data cleaning rules according to the rule configuration content.

[0109] In one embodiment, the data cleaning module 205 is specifically used to use a graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network; use the cleaning rules to formulate downstream associated field cleaning rules for the downstream associated fields; add the downstream associated field cleaning rules to the cleaning rule library to obtain a target cleaning rule library; and clean the target data according to the target cleaning rule library to generate cleaned data.

[0110] In one embodiment, the data cleaning module 205 is specifically used to use a depth-first search algorithm to analyze the association graph network to obtain the data association relationship of the target data; and find the downstream association field of the source layer data according to the data association relationship.

[0111] In one embodiment, the data cleaning module 205 is specifically used to, when receiving a target data segment to be cleaned, search for a target cleaning rule for the target data segment in a target cleaning rule library; determine a cleaning method for the target data segment according to the cleaning rule; clean the target data segment using the cleaning method according to the cleaning rule; and generate cleaned data after cleaning of several target data segments.

[0112] The embodiment of the present application also discloses a computer device.

[0113] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and execute the above-mentioned graph algorithm-based data cleaning method.

[0114] The embodiment of the present application also discloses a computer-readable storage medium.

[0115] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and execute a data cleaning method based on a graph algorithm as described above. The computer-readable storage medium includes, for example: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0116] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A data cleaning method based on graph algorithm, characterized in that: The method comprises: receiving target data to be cleaned, and collecting blood relationship information of the target data; Generate a correlation graph network of the target data according to the blood relationship information; Using a graph algorithm to find the source layer data of the target data, and formulating cleaning rules for the source layer data; Storing the plurality of cleaning rules in a preset cleaning rule library; The target data is cleaned according to the cleaning rule base and the association graph network.

2. The method according to claim 1, characterized in that: The graph algorithm includes a graph traversal algorithm and a conditional screening algorithm. The use of the graph algorithm to search for the source layer data of the target data includes: Analyzing the association graph network using the graph traversal algorithm to obtain an analysis result; Formulate screening conditions for the source layer data according to the analysis results; The source layer data of the target data is obtained by filtering according to the filtering conditions using a conditional filtering algorithm.

3. The method according to claim 1, characterized in that: The formulating of the cleaning rules of the source layer data includes: Determine the rule configuration content according to the source layer data, wherein the rule configuration content includes the rule type and the rule content; A cleaning rule for the source layer data is formulated according to the rule configuration content.

4. The method according to claim 1, characterized in that: The cleaning of the target data according to the cleaning rule base and the associated graph network comprises: Using a graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network; formulating a downstream associated field cleaning rule for the downstream associated field using the cleaning rule; Adding the downstream associated field cleaning rule to the cleaning rule base to obtain a target cleaning rule base; The target data is cleansed according to the target cleaning rule base to generate cleansed data.

5. The method according to claim 4, characterized in that: The graph algorithm includes a depth-first search algorithm, and the use of the graph algorithm to find the downstream associated fields of the source layer data according to the associated graph network includes: Analyzing the association graph network using the depth-first search algorithm to obtain the data association relationship of the target data; The downstream associated field of the source layer data is found according to the data association relationship.

6. The method according to claim 4, characterized in that: The target data includes a plurality of target data segments, and cleaning the target data according to the target cleaning rule base includes: When a target data segment to be cleaned is received, searching the target cleaning rule for the target data segment in the target cleaning rule library; Determining a cleaning method for the target data segment according to the cleaning rule; Cleaning the target data segment using the cleaning method according to the cleaning rule; When the cleaning of the target data segments is completed, cleaned data is generated.

7. The method according to claim 4, characterized in that: After the target data is cleaned according to the target cleaning rule base to generate cleaned data, the method further includes: Using a data quality check tool to verify the cleaned data to obtain a verification result, and determining whether the verification result is passed; If the verification result is passed, the data cleaning is successful; Otherwise, data cleaning fails and data cleaning is performed again.

8. A data cleaning device based on graph algorithm, characterized in that: The device comprises: A blood relationship information collection module (201) is used to receive target data to be cleaned and collect blood relationship information of the target data; A correlation graph network generation module (202), used to generate a correlation graph network of the target data according to the blood relationship information; A source layer data processing module (203), used for searching the source layer data of the target data by using a graph algorithm and formulating a cleaning rule for the source layer data; A cleaning rule base generation module (204), used for storing a plurality of the cleaning rules in a preset cleaning rule base; A data cleaning module (205) is used to clean the target data according to the cleaning rule base and the association graph network.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program which can be loaded by the processor and executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 7.

Images