A data quality control method, system, device and storage medium

Abstract

The present application relates to the technical field of data quality control, and in particular to a data quality control method, system, device and storage medium. The present application first collects order data and user data from an e-commerce platform, and divides them into multiple data partitions according to preset rules; then, each data partition is evaluated for quality, and the partition with quality problems is determined; for the problem data in the problem partition, the problem data is labeled and its quality problem type is identified, and then a quality control knowledge graph is constructed, which visualizes the entities corresponding to the order data, user data and quality problem types through entity nodes and relationship links, and displays the associations between them; it not only visualizes the data entities and their associations, but also can form accumulative quality control knowledge, which provides knowledge support and decision-making guidance for data quality control, and can better recognize and manage complex data quality issues, thereby improving the intelligence level of data quality control.

Description

Technical Field

[0001] The present application relates to the technical field of data quality control, and in particular to a data quality control method, system, device and storage medium. Background Art

[0002] With the rapid development of e-commerce, e-commerce platforms have accumulated a large amount of order data and user data. These data are not only an important support for e-commerce business operations, but also a valuable resource for exploring user needs and optimizing product marketing strategies. However, due to the diverse sources and complex formats of data, various quality problems are prone to occur, such as inconsistent and incomplete data, which affect the availability and accuracy of data. Therefore, effective data quality control is crucial to improving the data quality of e-commerce platforms and ensuring the scientific nature of business decisions.

[0003] Existing technologies usually use rule engines or automated methods based on machine learning to detect and evaluate data quality. Some technologies use metadata to manage data quality and establish data quality dimensional models; other technologies are based on big data platforms and use distributed computing to achieve large-scale data quality analysis. These methods can automatically identify data quality issues, but lack systematic management of quality issues, and this situation needs to be further improved. Summary of the invention

[0004] In order to solve the problem that existing data quality control lacks systematic management of quality issues, this application provides a data quality control method, system, device and storage medium, which adopts the following technical solutions:

[0005] In a first aspect, the present application provides a data quality control method, comprising the following steps:

[0006] Collect order data and user data from e-commerce platforms;

[0007] Based on a preset data partitioning rule, the order data and user data are divided into a plurality of data partitions;

[0008] Perform quality assessment on the order data and user data of each data partition, and determine the data partitions with quality problems based on the quality assessment results;

[0009] Marking the problematic data in the data partition with quality problems, and identifying the quality problem type of the problematic data based on the marking result;

[0010] Based on the identified quality problem types, a quality control knowledge graph is constructed. The knowledge graph visualizes the entities corresponding to the order data, the entities corresponding to the user data, and the entities corresponding to the quality problem types, as well as the relationships between them, through entity nodes and relationship links, to provide knowledge support for quality control decisions.

[0011] By adopting the above technical solution, this application first collects the order data and user data of the e-commerce platform, and divides them into multiple data partitions according to preset rules; then performs quality assessment on each data partition to determine the partition with quality problems; for the problem data in the problem partition, mark and identify its quality problem type, and then construct a quality control knowledge graph, visually display the order data, user data and entities corresponding to the quality problem type through entity nodes and relationship links, and display the associations between them; not only visually display data entities and their associations, but also can form accumulative quality control knowledge, provide knowledge support and decision-making guidance for data quality control, and can better recognize and manage complex data quality problems, thereby improving the intelligence level of data quality control.

[0012] Optionally, based on a preset data partitioning rule, the order data and user data are divided into a plurality of data partitions, specifically comprising the following steps:

[0013] Determine the order amount attribute as the partition attribute;

[0014] Based on a preset order amount interval threshold, the order data is divided into at least two order data partitions, each order data partition corresponding to an order amount interval;

[0015] In the case where the order amount changes, a change order is obtained, and the change order is re-divided into regions;

[0016] When a user is associated with multiple order data, a target user whose associated multiple order data are distributed in different order amount ranges is obtained, and a copy of the target user is saved to all associated order data partitions.

[0017] By adopting the above technical solution, the present application uses the order amount as a partitioning attribute, and first divides the order data into at least two order data partitions according to a preset order amount range threshold, and each partition corresponds to an order amount range; when the order amount changes, the system will obtain the changed order and re-divide the area for it; in addition, for the case where a user is associated with multiple order data and these order data are distributed in different amount ranges, the system will obtain the target user and save a copy of it to all relevant order data partitions; thereby reflecting the characteristic differences of data in different order amount ranges, which is conducive to improving the accuracy of subsequent quality assessment; at the same time, through the design of tracking order amount changes and retaining user copies, the dynamic nature and data integrity of partition division are ensured.

[0018] Optionally, the problematic data in the data partition with quality problems is labeled, and the quality problem type of the problematic data is identified based on the labeling result, which specifically includes the following steps:

[0019] Determine the annotation standards for different types of quality issues based on pre-built annotation rule sets;

[0020] From the data partitions determined to have quality problems, obtain problematic data as annotation objects;

[0021] Based on the standard rule set, automatically annotate the acquired problem data and output an initial annotation set;

[0022] Performing standardization processing on the initial annotation set, and scanning the standardized initial annotation set for quality problems based on a preset quality problem identification model to identify existing quality problems;

[0023] Based on the marking standard, the identified quality problem is associated with the corresponding quality problem type to determine the quality problem type corresponding to the quality problem.

[0024] By adopting the above technical scheme, the present application determines the labeling standards for different types of quality problems based on a pre-constructed labeling rule set, and standardizes the definition of quality problems; then obtains problem data from the data partitions with determined quality problems as labeling objects, and automatically labels these problem data based on the standard rule set, and outputs an initial labeling set; performs standardization processing on the initial labeling set, and then scans the standardized labeling set for quality problems based on a preset quality problem identification model to identify existing quality problems; realizes the automation and intelligence of problem data labeling and quality problem type identification; finally, according to the labeling standard, associates the identified quality problems with the corresponding quality problem types, and more accurately and efficiently identifies the types of quality problems, which helps to comprehensively and systematically understand and manage complex data quality problems.

[0025] Optionally, the method further comprises the following steps:

[0026] Sending the identified quality problem type to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type to obtain corrected data;

[0027] Receive the revised data, perform a quality check on the revised data, and if the revised data is of qualified quality, update the corresponding entities and association relationships in the quality control knowledge graph.

[0028] By adopting the above technical solution, after identifying the type of data quality problem, this application will feed back the information to the data source, and the data source will perform corresponding repair operations on the original data according to the type of quality problem to obtain corrected data; the system will receive these corrected data and perform quality checks on them to ensure that the corrected data meets the quality standards, and then update the corresponding entities and associations in the quality control knowledge graph; on the one hand, sending the quality problem type to the data source can help the data source accurately locate the data objects and repair methods that need to be repaired, thereby improving the repair efficiency. On the other hand, quality checks are performed on the corrected data to ensure the reliability of quality control. Only corrected data that meets the standards will be updated to the knowledge graph, ensuring the data quality of the knowledge graph.

[0029] Optionally, after receiving the revised data, performing a quality check on the revised data, and updating the corresponding entities and associations in the quality control knowledge graph if the revised data is of qualified quality, the method further comprises the following steps:

[0030] Performing a quality check on the corrected data, and determining a quality rating score of the corrected data partition based on a result of the quality check;

[0031] Compare the corrected data partition quality rating scores with the data partition quality rating scores before correction to identify improvements in data quality;

[0032] Add the improvement of the data quality to the corresponding data partition entity in the quality control knowledge graph to demonstrate the data repair effect;

[0033] Based on the updated quality rating scores of each data partition, the average overall quality rating scores of all data partitions are determined.

[0034] By adopting the above technical solution, after updating the knowledge graph, this application performs a quality check on the corrected data to determine the quality rating score of the corrected data partition; then compares the corrected score with the score before correction to identify the improvement in data quality, and adds this improvement to the corresponding data partition entity to intuitively display the data repair effect; finally, based on the updated rating scores of each partition, calculates the overall quality rating average of all partitions; achieves comprehensive tracking and presentation of data quality improvements; the comparison of rating scores directly reflects the quality improvement brought about by data repair, and visualizes the improvement in the knowledge graph; quality control decision makers can clearly understand the contribution of repair behavior to improving data quality, and judge whether the quality improvement effect meets expectations.

[0035] Optionally, the quality problem type includes a status identifier;

[0036] The identified quality problem type is sent to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type and sets the status flag to "repair pending confirmation";

[0037] After receiving the correction data and performing a quality check on the correction data, if the correction data is of qualified quality, setting the status flag to "repaired".

[0038] By adopting the above technical solution, this application introduces status identification for quality problem types to track the repair status of quality problems, which helps to better control and manage the repair process; enables the system to accurately grasp the repair progress of each quality problem to avoid omissions or repeated repairs; secondly, the entire life cycle of the quality problem can be restored through state changes, providing a basis for subsequent analysis and optimization.

[0039] Optionally, before sending the identified quality problem type to the data source end so that the data source end performs a repair operation on the original data according to the quality problem type to obtain the corrected data, the method further includes the following steps:

[0040] Based on the quality control knowledge graph, a visual quality problem distribution report is generated to intuitively display the types and corresponding quantities of quality problems existing in each data partition;

[0041] Receive quality control measures input by a user, wherein the quality control strategy includes quality problem priority information and corresponding processing methods;

[0042] Based on the quality problem distribution report and the quality control strategy, determine the types of quality problems to be repaired and the priority ranking of their corresponding data partitions;

[0043] A repair operation is performed on the original data according to the priority order.

[0044] By adopting the above technical solution, this application first generates a visual quality problem distribution report based on the quality control knowledge graph, intuitively displaying the types and quantities of quality problems existing in each data partition; then receives the quality control strategy input by the user, including quality problem priority information and corresponding processing methods; the system determines the type of quality problem and the data partition in which it is located that need to be repaired first based on the problem distribution report and the control strategy, and generates a priority ranking, and finally performs repair operations on the original data in sequence according to this ranking; it realizes the systematic planning and phased advancement of data repair work, which helps to reasonably allocate repair resources and improve repair efficiency; at the same time, the manually set control strategy is introduced into the decision-making link, which meets the actual business needs and ensures the targeted priority ranking.

[0045] In a second aspect, the present application provides a data quality control system, comprising:

[0046] Data collection module, used to collect order data and user data from e-commerce platforms;

[0047] A data partitioning module, used to partition the order data and user data into multiple data partitions based on a preset data partitioning rule;

[0048] A quality assessment module is used to perform quality assessment on the order data and user data of each data partition, and determine the data partition with quality problems based on the quality assessment results;

[0049] A data labeling module, used to label the problematic data in the data partition with quality problems, and identify the quality problem type of the problematic data based on the labeling result;

[0050] A graph construction module is used to construct a quality control knowledge graph based on the identified quality problem types. The knowledge graph visually displays the entities corresponding to the order data, the entities corresponding to the user data, and the entities corresponding to the quality problem types and their mutual relationships through entity nodes and relationship links, thereby providing knowledge support for quality control decisions.

[0051] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned data quality control method when executing the computer program.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned data quality control method when executed by a processor.

[0053] In summary, the present application includes at least one of the following beneficial technical effects:

[0054] 1. This application first collects order data and user data from the e-commerce platform, and divides them into multiple data partitions according to preset rules; then conducts quality assessment on each data partition to determine the partition with quality problems; annotates the problem data in the problem partition and identifies its quality problem type, and then constructs a quality control knowledge graph, visually displays order data, user data, and entities corresponding to the quality problem type through entity nodes and relationship links, and displays the associations between them; not only visually displays data entities and their associations, but also can form accumulative quality control knowledge, provide knowledge support and decision-making guidance for data quality control, and can better recognize and manage complex data quality problems, thereby improving the intelligence level of data quality control;

[0055] 2. This application uses the order amount as a partitioning attribute. First, according to the preset order amount interval threshold, the order data is divided into at least two order data partitions, each partition corresponding to an order amount interval; when the order amount changes, the system will obtain the changed order and re-divide the area for it; in addition, in the case where a user is associated with multiple order data and these order data are distributed in different amount intervals, the system will obtain the target user and save its copy to all relevant order data partitions; thereby reflecting the characteristic differences of data in different order amount intervals, which is conducive to improving the accuracy of subsequent quality assessment; at the same time, by tracking the changes in order amounts and retaining user copies, the dynamic nature and data integrity of partition division are ensured;

[0056] 3. This application determines the labeling standards for different types of quality problems based on a pre-built labeling rule set, and defines quality problems in a standardized manner; then, problem data is obtained from the data partitions with determined quality problems as labeling objects, and these problem data are automatically labeled based on the standard rule set, and an initial labeling set is output; standardization is performed on the initial labeling set, and then the standardized labeling set is scanned for quality problems based on a preset quality problem identification model to identify existing quality problems; automation and intelligence of problem data labeling and quality problem type identification are achieved; finally, according to the labeling standard, the identified quality problems are associated with the corresponding quality problem types, and the types of quality problems are identified more accurately and efficiently, which helps to comprehensively and systematically understand and manage complex data quality problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a flow chart of a data quality control method according to an embodiment of the present application;

[0058] Figure 2 It is a flowchart of step S120 in a data quality control method in an embodiment of the present application;

[0059] Figure 3 It is a flowchart of step S140 in a data quality control method in an embodiment of the present application;

[0060] Figure 4 It is a schematic diagram of the process of correcting data in a data quality control method in an embodiment of the present application;

[0061] Figure 5 It is a flowchart of step S410 in a data quality control method in an embodiment of the present application;

[0062] Figure 6 This is a module diagram of a data quality control system according to an embodiment of the present application;

[0063] Figure 7It is a diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more listed items.

[0065] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0066] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0067] In the first aspect, the present application provides a data quality control method, referring to Figure 1 , the method comprises the following steps:

[0068] S110. Collect order data and user data from the e-commerce platform.

[0069] In this embodiment, order data and user data are usually stored in the database system of the e-commerce platform and need to be collected from the database system. Order data generally includes fields such as order number, order time, order amount, and consignee information; user data generally includes fields such as user ID, user name, contact information, and registration time. These data can be obtained using database query statements or data collection tools.

[0070] Specifically, the database query language is used to obtain the data of the required fields according to the structure definition of the order and user related tables.

[0071] S120. Divide the order data and user data into multiple data partitions based on a preset data partitioning rule.

[0072] In this embodiment, the data segmentation rules are pre-set by the data analyst, such as using the order amount as the basis for segmentation and setting multiple amount range thresholds; using the order time as the basis for segmentation and setting multiple time range thresholds; using the delivery address area as the basis for segmentation and setting multiple location range thresholds, etc.

[0073] Specifically, order data is partitioned according to the rules, such as order amounts of 0-99 yuan are assigned to one partition, and 100-499 yuan are assigned to another partition, etc. For user data, if a user is associated with multiple orders and these orders are distributed in partitions with different amounts, the user data will be copied and cannot be distributed to all corresponding partitions.

[0074] S130 , performing quality assessment on the order data and user data of each data partition, and determining the data partition with quality problems based on the quality assessment results.

[0075] In this embodiment, different quality rule sets can be designed for different data partitions, and the evaluation criteria can be customized according to business requirements.

[0076] Specifically, for each data partition, a different set of quality rules is designed to check whether the order data and user data comply with the rules. For example, the rules for the order amount partition include: whether the order amount is within the specified range, whether the amount field is a number, etc. Use automated tools such as rule engines to traverse the data of each partition to check whether it complies with the corresponding rules. For data partitions that are evaluated as unqualified, mark them as having quality issues.

[0077] S140: Label the problematic data in the data partition with quality problems, and identify the quality problem type of the problematic data based on the labeling result.

[0078] In this embodiment, the system pre-constructs a quality problem annotation rule set to standardize the definition of different types of quality problems as a basis for subsequent automatic annotation.

[0079] Specifically, define the annotation standards for different quality issues, such as the "format error" problem corresponding to the non-numeric amount field. Then extract the problem data from the problem partition, automatically annotate based on the annotation rules, and output the initial annotation set. Perform data cleaning and standardization on the annotation set, and then use the quality problem identification model, such as the classification model based on machine learning, to scan the standardized annotation set to identify the specific types of quality problems, such as "format error" and "missing data". Finally, according to the annotation standards, associate the identified quality problems with the quality problem types.

[0080] S150. Construct a quality control knowledge graph based on the identified quality problem types.

[0081] Among them, the knowledge graph visualizes the entities corresponding to order data, entities corresponding to user data, entities corresponding to quality problem types, and the relationships between them through entity nodes and relationship links, providing knowledge support for quality control decisions.

[0082] In this embodiment, a quality control knowledge graph is constructed through a graph database.

[0083] Specifically, entities corresponding to order data, user data, and quality problem types are added as nodes in the knowledge graph. The knowledge graph construction tool is used to automatically generate relationship links based on the relationships between entities, such as orders associated with users, orders with quality problems, etc., to display the associations between entity nodes, and finally form a visual knowledge graph that displays entity nodes and their associated networks.

[0084] In one embodiment, referring to Figure 2 In step S120, based on a preset data partitioning rule, the order data and user data are divided into multiple data partitions, which specifically includes the following steps:

[0085] S121. Determine the order amount attribute as a partitioning attribute.

[0086] In this embodiment, the order amount is selected as the classification attribute. The order amount, as an important attribute of the order data, can better reflect the value and level of the order.

[0087] Specifically, order amount data is usually stored in the amount field of the order table. This application obtains order amount information from the amount field and classifies orders according to the value of the order amount.

[0088] S122. Based on a preset order amount interval threshold, divide the order data into at least two order data partitions, each order data partition corresponding to an order amount interval.

[0089] In this embodiment, the setting of the order amount interval threshold can be customized according to the actual business situation and needs of the enterprise, and different amount intervals represent orders of different levels.

[0090] Specifically, the order data is divided into corresponding partitions according to the range of order amounts. Each partition contains all order records within the amount range, thereby performing hierarchical management of order data.

[0091] S123. When the order amount changes, obtain the change order and re-divide the area for the change order.

[0092] In this embodiment, the order amount may change due to certain reasons, such as discounts, additional orders, etc. At this time, the changed order needs to be re-partitioned.

[0093] Specifically, this application sets up an order amount change monitoring mechanism. Once a change in the order amount is found, the changed order will be extracted, and the amount partition to which it should belong will be recalculated based on the changed amount, and the order data will be migrated to the correct partition, thereby ensuring the consistency and accuracy of the data within the partition.

[0094] S124. When a user is associated with multiple order data, a target user whose associated multiple order data are distributed in different order amount ranges is obtained, and a copy of the target user is saved to all associated order data partitions.

[0095] In this embodiment, a user is associated with multiple orders. Since the orders may be distributed in different amount partitions, it is necessary to ensure the integrity of the user data in all related partitions.

[0096] Specifically, by traversing all users, obtaining the order data associated with each user, and checking whether the order data is distributed in multiple different amount partitions, if so, saving copies of the user data in all related partitions; thus, all order information of a certain user is taken into account during analysis, and some related data will not be lost due to partition division.

[0097] In one embodiment, referring to Figure 3 In step S140, the problematic data in the data partition with quality problems are labeled, and the quality problem type of the problematic data is identified based on the labeling result, which specifically includes the following steps:

[0098] S141. Based on a pre-built annotation rule set, determine annotation standards for different types of quality issues.

[0099] S142. Obtain problematic data from the determined data partitions with quality problems as marking objects.

[0100] S143. Based on the standard rule set, the acquired problem data is automatically annotated and an initial annotation set is output.

[0101] S144, performing standardization processing on the initial annotation set, and based on a preset quality problem identification model, performing quality problem scanning on the standardized initial annotation set to identify existing quality problems.

[0102] In this embodiment, the initial annotation set after automatic annotation may contain some impurities and noise, so data cleaning and standardization processing are required to improve the accuracy of subsequent recognition.

[0103] Specifically, standardization operations such as unification of uppercase and lowercase letters, special character processing, and outlier filtering are performed on the initial annotation set to normalize the original data format. Then, based on a pre-trained quality problem identification model such as a machine learning classification model, the standardized annotation set is scanned. The model classifies and identifies quality problems for each piece of data based on the results of historical data training and outputs the identification results.

[0104] S145. Based on the labeling standard, the identified quality problem is associated with the corresponding quality problem type to determine the quality problem corresponding to the quality problem.

[0105] In this embodiment, after completing the standardization process and identifying the quality issues, the system associates and summarizes the results of the two steps.

[0106] Specifically, based on the previously defined annotation standards, a mapping relationship between quality issues and quality issue types is established. The automatically identified quality issue results are then associated with the corresponding data records in the original annotation set, and then matched with the quality issue type defined in the annotation rule set for the record, thereby determining the specific quality issue type corresponding to each quality issue instance.

[0107] In one embodiment, referring to Figure 4 , the method further comprises the following steps:

[0108] S410: Send the identified quality problem type to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type to obtain corrected data.

[0109] In this embodiment, the identified quality problem type is fed back to the data source end, so that the provider of the original data can understand the specific quality problem and perform data repair in a targeted manner.

[0110] Specifically, through the data quality feedback mechanism, different types of quality problems are sent to the system at the data source through a customized feedback interface; after receiving the feedback, the data source adopts different repair strategies according to the type of problem, such as filling in missing values, normalizing format errors, etc., so as to obtain a corrected data set that has been manually or automatically repaired.

[0111] S420, receiving the revised data, performing a quality check on the revised data, and updating the corresponding entities and relationships in the quality control knowledge graph if the revised data quality is qualified.

[0112] In this embodiment, after obtaining the corrected data, quality verification is performed to ensure that the corrected data meets the quality requirements, and finally applied to the quality control knowledge graph.

[0113] Specifically, by modifying the data quality assessment rules, the modified data is scanned line by line to check whether the modified data still has quality defects; for the modified data with qualified quality, it is updated to the corresponding data entity node in the knowledge graph, and the relevant association relationship links are adjusted according to the new data, so that the knowledge graph continues to reflect the latest data quality status. If the modified data is still unqualified, it will be fed back to the data source for further repair until the quality meets the standard.

[0114] Furthermore, the quality problem type includes status identification;

[0115] The identified quality problem type is sent to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type and then sets the status indicator to repair pending confirmation.

[0116] After receiving the correction data and performing a quality check on the correction data, if the quality of the correction data is qualified, the status indicator is set to repaired.

[0117] Specifically, a "status" attribute field is added to the quality problem type entity, and the value of this field is the status identifier. The initial status can be set to "unrepaired" or other default values. By traversing all quality problem type instances sent for repair, their "status" attribute values ​​are set to "repair pending confirmation", indicating that these problems have been fed back for repair but the repair results have not yet been confirmed. After updating the data entities and relationships in the knowledge graph, obtain the quality problem type instances involved, and set the "status" attribute value to "repaired", indicating that the problem has been resolved through this repair.

[0118] By introducing status identifiers, the repair progress of each quality issue can be more clearly depicted in the knowledge graph, providing more comprehensive support for subsequent quality analysis and decision-making. Developers do not need to check the repair records, but only need to pay attention to the status identifiers to understand the repair status of various issues.

[0119] Furthermore, in one embodiment, after step S420, the method further includes the following steps:

[0120] S430: Perform a quality check on the corrected data, and determine a quality rating score for the corrected data partition based on the result of the quality check.

[0121] In this embodiment, in addition to determining whether the corrected data is of qualified quality, the overall quality level of the corrected data partition is also quantitatively evaluated.

[0122] Specifically, a comprehensive quality check is performed on the corrected data partition, and a comprehensive score is calculated according to the check results and rules as the quality rating score of the data partition. The scoring rules include multiple dimensions such as data completeness, accuracy, and consistency.

[0123] S440: Compare the corrected data partition quality rating score with the data partition quality rating score before correction to identify improvements in data quality.

[0124] In this embodiment, by comparing the partition quality rating scores before and after the correction, it is determined whether and to what extent the data quality has been improved.

[0125] Specifically, the system saves the historical quality rating scores for each data partition. After calculating the revised new score, it is compared with the old score before the most recent revision. If the new score is higher than the old score, it indicates that the data quality has improved, and the score difference is the extent of the improvement.

[0126] S450. Add the improvement of data quality to the corresponding data partition entity in the quality control knowledge graph to demonstrate the data repair effect.

[0127] In this embodiment, the data quality improvement situation is integrated into the knowledge graph so that the graph not only shows the current quality status but also presents the quality improvement process.

[0128] Specifically, a "quality improvement" attribute can be added to the data partition entity, and the comparison result can be stored in the attribute, such as "the quality score has increased by 10 points compared to the last time." Through graphical display, users can see the data repair effect of each partition at a glance.

[0129] S460: Based on the updated quality rating scores of each data partition, determine the average overall quality rating score of all data partitions.

[0130] In this embodiment, the overall data quality level is reflected by the global average score.

[0131] Specifically, the latest quality rating scores of all data partitions are traversed, and the weighted average is calculated according to the weight rule, such as the proportion of partition data volume, as the average overall quality rating score of all partitions. Furthermore, the overall quality trend is analyzed by comparing the average overall quality rating score with the historical value.

[0132] In one embodiment, referring to Figure 5 In step S410, the identified quality problem type is sent to the data source end so that the data source end performs a repair operation on the original data according to the quality problem type. Before obtaining the corrected data, the method further includes the following steps:

[0133] S510. Based on the quality control knowledge graph, a visual quality problem distribution report is generated to intuitively display the types and corresponding quantities of quality problems existing in each data partition.

[0134] In this embodiment, the quality problem distribution report can enable decision makers to more quickly understand the current data quality status.

[0135] Specifically, various statistical charts can be generated based on the quality problem type entities, data partition entities and the relationship between the two in the knowledge graph, such as a bar chart showing the number of problems distributed by problem type in each partition, and a pie chart showing the proportion of each problem type in a partition. Through visual presentation, the distribution of quality problems in different partitions and different problem types can be intuitively reflected.

[0136] S520: Receive quality control measures input by the user, where the quality control strategy includes quality problem priority information and corresponding processing methods.

[0137] In this embodiment, the user sets priorities and corresponding processing strategies for different quality issues according to actual needs.

[0138] Specifically, this application provides an interactive interface that allows users to select the type of quality issues they are concerned about, and set priorities for each type, such as high, medium, and low, and the desired processing methods, such as repair, monitoring, etc. For example, for "data missing" type issues, users can set a high priority and require the system to perform repair operations first.

[0139] S530: Based on the quality problem distribution report and the quality control strategy, determine the types of quality problems that need to be repaired and the priority ranking of their corresponding data partitions.

[0140] In this embodiment, the system comprehensively considers the distribution situation and the management and control strategy, and intelligently determines the priority of repair.

[0141] Specifically, traverse each quality problem type, obtain the distribution quantity of each quality problem type in each partition, assign a weight value according to the priority level set by the user, and calculate the comprehensive weight score of the problem type. Sort all problem types from high to low according to the comprehensive score, and the top types and corresponding partitions are given priority repair objects.

[0142] S540: Perform a repair operation on the original data according to the priority order.

[0143] In this embodiment, the system performs data repair work according to the determined priority.

[0144] Specifically, starting from the first problem type in the sorting results, locate the data partition where it is located, read the original data, and execute data repair strategies such as rule modification and manual review according to the characteristics of the problem type until the corresponding type of problems are all repaired in these partitions. Then process the next priority problem type and perform the repair operation in a loop until all problem types within the priority range are repaired.

[0145] The implementation principle of the embodiment of the present application is: first, a visual quality problem distribution report is generated based on the quality control knowledge graph to intuitively display the types and quantities of quality problems existing in each data partition; then the quality control strategy input by the user is received, including quality problem priority information and corresponding processing methods; the system determines the type of quality problem that needs to be repaired first and the data partition in which it is located based on the problem distribution report and the control strategy, and generates a priority ranking, and finally performs repair operations on the original data in sequence according to this ranking; it realizes the systematic planning and phased advancement of data repair work, which helps to reasonably allocate repair resources and improve repair efficiency; at the same time, the manually set control strategy is introduced into the decision-making link, which meets the actual business needs and ensures the targeted priority ranking.

[0146] In a second aspect, the present application provides a data quality control system. The data quality control system of the present application is described below in combination with the above-mentioned data quality control method.

[0147] Reference Figure 6 , a data quality control system, comprising:

[0148] Data collection module 610, used to collect order data and user data of the e-commerce platform;

[0149] A data partitioning module 620, configured to partition the order data and the user data into a plurality of data partitions based on a preset data partitioning rule;

[0150] A quality assessment module 630, configured to perform quality assessment on the order data and user data of each data partition, and determine the data partition with quality problems based on the quality assessment result;

[0151] A data labeling module 640 is used to label the problematic data in the data partition with quality problems, and identify the quality problem type of the problematic data based on the labeling result;

[0152] The graph construction module 650 is used to construct a quality control knowledge graph based on the identified quality problem types. The knowledge graph visualizes the entities corresponding to the order data, the entities corresponding to the user data, and the entities corresponding to the quality problem types and their mutual relationships through entity nodes and relationship links, thereby providing knowledge support for quality control decisions.

[0153] In one embodiment, the present application provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown. The electronic device includes a processor, a memory and a network interface connected via a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a data quality control method is implemented.

[0154] Those skilled in the art will understand that Figure 7 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0156] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0157] The above are all preferred embodiments of the present application, and the protection scope of the present application is not limited thereto. Therefore, any equivalent changes made according to the structure, shape, and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data quality control method, characterized in that: The steps include: Collect order data and user data from e-commerce platforms; Based on a preset data partitioning rule, the order data and user data are divided into a plurality of data partitions; Perform quality assessment on the order data and user data of each data partition, and determine the data partitions with quality problems based on the quality assessment results; Marking the problematic data in the data partition with quality problems, and identifying the quality problem type of the problematic data based on the marking result; Based on the identified quality problem types, a quality control knowledge graph is constructed. The knowledge graph visually displays the entities corresponding to the order data, the entities corresponding to the user data, and the entities corresponding to the quality problem types and their mutual relationships through entity nodes and relationship links, providing knowledge support for quality control decisions; Wherein, based on the preset data partitioning rule, the order data and user data are divided into multiple data partitions, which specifically includes the following steps: Determine the order amount attribute as the partition attribute; Based on a preset order amount interval threshold, the order data is divided into at least two order data partitions, each order data partition corresponding to an order amount interval; In the case where the order amount changes, a change order is obtained, and the change order is re-divided into regions; When a user is associated with multiple order data, a target user whose associated multiple order data are distributed in different order amount ranges is obtained, and a copy of the target user is saved to all associated order data partitions; The problem data in the data partition with quality problems are marked, and the quality problem type of the problem data is identified based on the marking result, which specifically includes the following steps: Determine the annotation standards for different types of quality issues based on pre-built annotation rule sets; From the data partitions determined to have quality problems, obtain problematic data as annotation objects; Based on the annotation rule set, automatically annotate the acquired problem data and output an initial annotation set; Performing standardization processing on the initial annotation set, and scanning the standardized initial annotation set for quality problems based on a preset quality problem identification model to identify existing quality problems; Based on the marking standard, the identified quality problem is associated with the corresponding quality problem type to determine the quality problem type corresponding to the quality problem.

2. The data quality control method according to claim 1, characterized in that: The method further comprises the steps of: Sending the identified quality problem type to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type to obtain corrected data; Receive the revised data, perform a quality check on the revised data, and if the revised data is of qualified quality, update the corresponding entities and association relationships in the quality control knowledge graph.

3. The data quality control method according to claim 2, characterized in that: After receiving the revised data, performing a quality check on the revised data, and updating the corresponding entities and associations in the quality control knowledge graph if the revised data is of qualified quality, the method further includes the following steps: Performing a quality check on the revised data, and determining a quality rating score of the revised data partition based on a result of the quality check; Compare the corrected data partition quality rating scores with the data partition quality rating scores before correction to identify improvements in data quality; Add the improvement of the data quality to the corresponding data partition entity in the quality control knowledge graph to demonstrate the data repair effect; Based on the updated quality rating scores of each data partition, the average overall quality rating scores of all data partitions are determined.

4. The data quality control method according to claim 2, characterized in that: The quality problem type includes a status identifier; The identified quality problem type is sent to the data source end, so that the data source end performs a repair operation on the original data according to the quality problem type and sets the status flag to "repair pending confirmation"; After receiving the correction data and performing a quality check on the correction data, if the correction data is of qualified quality, setting the status flag to "repaired".

5. The data quality control method according to claim 2, characterized in that: The method further includes the following steps before sending the identified quality problem type to the data source end so that the data source end performs a repair operation on the original data according to the quality problem type to obtain the corrected data: Based on the quality control knowledge graph, a visual quality problem distribution report is generated to intuitively display the types and corresponding quantities of quality problems existing in each data partition; Receive a quality control strategy input by a user, wherein the quality control strategy includes quality problem priority information and corresponding processing methods; Based on the quality problem distribution report and the quality control strategy, determine the types of quality problems to be repaired and the priority ranking of their corresponding data partitions; A repair operation is performed on the original data according to the priority order.

6. A data quality control system, characterized in that: include: Data collection module, used to collect order data and user data from e-commerce platforms; A data partitioning module, used to partition the order data and user data into multiple data partitions based on a preset data partitioning rule; A quality assessment module is used to perform quality assessment on the order data and user data of each data partition, and determine the data partition with quality problems based on the quality assessment results; A data labeling module, used to label the problematic data in the data partition with quality problems, and identify the quality problem type of the problematic data based on the labeling result; A graph construction module is used to construct a quality control knowledge graph based on the identified quality problem types. The knowledge graph visually displays the entities corresponding to the order data, the entities corresponding to the user data, and the entities corresponding to the quality problem types and their mutual relationships through entity nodes and relationship links, so as to provide knowledge support for quality control decisions; The data partitioning module performs the following steps: Determine the order amount attribute as the partition attribute; Based on a preset order amount interval threshold, the order data is divided into at least two order data partitions, each order data partition corresponding to an order amount interval; In the case where the order amount changes, a change order is obtained, and the change order is re-divided into regions; When a user is associated with multiple order data, a target user whose associated multiple order data are distributed in different order amount ranges is obtained, and a copy of the target user is saved to all associated order data partitions; The data annotation module performs the following steps: Determine the annotation standards for different types of quality issues based on pre-built annotation rule sets; From the data partitions determined to have quality problems, obtain problematic data as annotation objects; Based on the annotation rule set, automatically annotate the acquired problem data and output an initial annotation set; Performing standardization processing on the initial annotation set, and scanning the standardized initial annotation set for quality problems based on a preset quality problem identification model to identify existing quality problems; Based on the marking standard, the identified quality problem is associated with the corresponding quality problem type to determine the quality problem type corresponding to the quality problem.

7. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the data quality control method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of data quality control according to any one of claims 1 to 5 are implemented.

Images