A machine learning based data quality management method and system

Abstract

The application relates to a kind of data quality management method and system based on machine learning, belong to data processing technical field. Including: obtaining data to be processed;Determine the comprehensive data quality score of data to be processed;According to the comprehensive data quality score, determine the abnormal level of data to be processed;According to the abnormal level, respectively, direct trigger automation repair, batch timing repair and record generation are carried out, and repair is carried out after subsequent manual review;When repairing data to be processed, according to the data repair rule pre-configured by the system, the repair strategy of data to be processed is determined through the naive Bayes classifier;Repair strategy is used to repair data to be processed;Judge whether the cleanness of repaired data is greater than the preset cleanness;If yes, complete data repair;Otherwise, issue an alarm, and wait for manual repair.The application improves the accuracy of data quality evaluation, reduces the need for manual intervention, and ensures that the data repair process is more efficient and accurate.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a data quality management method and system based on machine learning. Background Technology

[0002] Machine learning-based data quality management is an automated process that uses machine learning algorithms to detect, evaluate, and repair data quality. It can handle large-scale datasets and detect data quality problems in real time, thereby improving the accuracy, integrity, and consistency of data and providing higher-quality support for data-driven decision-making.

[0003] With the explosive growth in data volume and increasing complexity, traditional manual inspection and rule-driven data quality management methods are no longer sufficient to meet the challenges of the modern data environment. Machine learning technology can automatically identify and correct errors, anomalies, and inconsistencies in data, significantly improving the efficiency and accuracy of data quality management, thereby providing enterprises with more reliable decision support and stronger data analysis capabilities.

[0004] However, traditional data repair usually relies on manual review and intervention. Manual intervention makes the repair process inefficient, introduces human error, and affects the overall quality of the data. Traditional data repair strategies are usually static and cannot be dynamically adjusted according to the actual situation of the data. Traditional data quality assessment methods often do not systematically assess data quality and lack the ability to perform quality checks on data integrity, accuracy, relevance, and other aspects. This makes it impossible for the data cleaning process to accurately locate the root cause of the problem and for the repair work to be comprehensive. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the purpose of the invention is to provide a data quality management method and system based on machine learning, which can improve the accuracy of data quality assessment, reduce the need for manual intervention, and ensure that the data repair process is more efficient and accurate.

[0006] In a first aspect, the present invention proposes a data quality management method based on machine learning, the method comprising:

[0007] S1, Obtain the data to be processed;

[0008] S2, perform data quality detection on the data to be processed, and determine the comprehensive data quality score of the data to be processed;

[0009] S3. Based on the comprehensive data quality score, determine the anomaly level of the data to be processed, wherein the anomaly level includes critical anomaly, general anomaly, and minor anomaly.

[0010] S4, when the data to be processed is urgent abnormal data, automatic repair is triggered directly; when the data to be processed is general abnormal data, batch timed repair is performed; when the data to be processed is minor abnormal data, a record is generated, and repair is performed after subsequent manual review.

[0011] S5. When repairing the data to be processed, the repair strategy for the data to be processed is determined by using a Naive Bayes classifier according to the data repair rules pre-configured by the system.

[0012] S6, The repair strategy is used to repair the data to be processed;

[0013] S7, determine whether the cleanliness of the data to be processed after repair is greater than the preset cleanliness; if yes, complete the data repair; otherwise, issue an alarm and wait for manual repair.

[0014] Furthermore, S2 specifically includes:

[0015] S201, Perform integrity detection on the data to be processed and determine the integrity score;

[0016] S202, perform correlation detection on the data to be processed and determine the correlation score;

[0017] S203, perform accuracy detection on the data to be processed and determine the accuracy score;

[0018] S204, combining the integrity score, the relevance score, and the accuracy score, determine the comprehensive data quality score of the data to be processed.

[0019] Further, S201 specifically includes:

[0020] S2011, Based on the data to be processed, determine the integrity data unit, wherein the integrity data unit includes: column, tuple and cell;

[0021] S2012, Determine the integrity score based on the integrity data unit.

[0022] Furthermore, S202 specifically includes:

[0023] S2021, Based on the data to be processed, determine the relevant data units, wherein the relevant data units include: databases, tables, tuples, and columns;

[0024] S2022, Determine the correlation score based on the correlation data unit.

[0025] Furthermore, S203 specifically includes:

[0026] S2031, determine the data type of the data to be processed, wherein the data type includes: character type and numeric type;

[0027] S2032, Determine the accuracy score based on the data type.

[0028] Furthermore, the repair strategy specifically includes: an RHS repair strategy and an LHS repair strategy.

[0029] Furthermore, S5 specifically includes:

[0030] S501, determine the prior probability and conditional probability of each of the repair strategies;

[0031] S502, calculate the posterior probability of each of the repair strategies based on the prior probability and the conditional probability;

[0032] S503, determine the repair strategy for the data to be processed based on the posterior probability.

[0033] Furthermore, S503 specifically includes:

[0034] When the posterior probability of the RHS repair strategy is higher than that of the LHS repair strategy, the repair strategy is the RHS repair strategy.

[0035] When the posterior probability of the RHS repair strategy is lower than the posterior probability of the LHS repair strategy, the repair strategy is the LHS repair strategy.

[0036] The beneficial effects of this invention are as follows:

[0037] In this embodiment of the invention, by calculating a comprehensive data quality score, data is categorized into urgent anomalies, general anomalies, and minor anomalies. Different repair methods are adopted according to the anomaly level, effectively improving the accuracy of data quality assessment, reducing resource waste, and enhancing the targeting and effectiveness of data repair. The repair strategy is automatically predicted through the data quality score, thereby achieving automated data repair and avoiding the errors and inefficiencies that may occur during manual repair. This significantly improves repair efficiency. A Naive Bayes classifier is used to calculate the posterior probability of each repair strategy, dynamically selecting the optimal strategy, making the repair process more intelligent. Optimal repair is performed based on the context of different data. By calculating data cleanliness and comparing it with a preset cleanliness threshold, the repair effect can be dynamically judged, ensuring that the data ultimately meets the expected quality standards. This invention, through machine learning technology and automated repair strategies, reduces the need for manual intervention, ensuring a more efficient and accurate data repair process. Attached Figure Description

[0038] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.

[0039] Figure 1 This is a flowchart illustrating a data quality management method based on machine learning provided in an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a machine learning-based data quality management system provided in an embodiment of the present invention. Detailed Implementation

[0041] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0042] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.

[0043] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0044] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.

[0045] This invention proposes a data quality management method and system based on machine learning, which addresses the problems of cold start, data sparsity, and relatively simple recommendations in traditional recommendation methods, leading to a decline in user experience and the effectiveness of recommendation results, and thus failing to better adapt to changes in users' personalized needs.

[0046] Method Implementation Examples

[0047] Reference Figure 1 The diagram illustrates a flowchart of a data quality management method based on machine learning provided in an embodiment of the present invention.

[0048] This invention provides a data quality management method based on machine learning, the method comprising:

[0049] Specifically, the method includes steps S1 to S7.

[0050] S1, Obtain the data to be processed.

[0051] S2, perform data quality testing on the data to be processed, and determine the overall data quality score of the data to be processed.

[0052] It should be noted that by conducting data quality inspections and providing comprehensive data quality scores, we can systematically evaluate the quality of various aspects of the data, thereby helping to identify potential problems and risks and ensuring that subsequent analysis and decision-making are based on high-quality data.

[0053] In one possible implementation, S2 specifically includes:

[0054] S201, Perform integrity checks on the data to be processed and determine the integrity score.

[0055] Integrity testing refers to checking whether data is missing or contains all necessary information to ensure data integrity.

[0056] In one possible implementation, S201 specifically includes:

[0057] S2011, Based on the data to be processed, determine the integrity data unit, wherein the integrity data unit includes: column, tuple and cell.

[0058] In this context, a column refers to a field in a data table, typically representing a specific attribute (such as name or age). A tuple is a row in a data table, representing a complete record. A cell is a specific data point in the data table, where columns and tuples intersect, representing the specific value of a certain attribute in a particular record.

[0059] Specifically, by dividing the data to be processed into complete data units, missing or incomplete parts of the data can be accurately located and corrected in a targeted manner, which can improve the accuracy of data quality detection and make the data cleaning process more efficient and operable.

[0060] S2012, Determine the integrity score based on the integrity data unit.

[0061] Specifically, determining an integrity score based on integrity data units can systematically quantify the degree of missing or incomplete parts in the data, provide targeted data correction directions, accurately identify data quality problems, improve data integrity, and ensure the reliability of subsequent analysis and decision-making.

[0062] When the integrity data unit is a column, the integrity score is as follows:

[0063]

[0064] Where WZ represents the integrity score, and CW a This represents the weight of column a of the data to be processed, CC a This indicates the completeness of column a, that is, the proportion of non-missing values ​​in column a to the total value, where a = 1, 2, ..., n, and n represents the total number of columns of data to be processed.

[0065] When the integrity data unit is a tuple, the integrity score is as follows:

[0066]

[0067] Among them, TW j TC represents the weight of the j-th tuple. j This represents the completeness of the j-th tuple, i.e., the proportion of non-missing values ​​in the j-th tuple to the total value, where j = 1, 2, ..., m, and m represents the total number of tuples in the data to be processed.

[0068] When the integrity data unit is a cell, the integrity score is as follows:

[0069]

[0070] Among them, Cell ja This represents the cell value at the intersection of the j-th tuple and the a-th column.

[0071] S202, Perform correlation detection on the data to be processed and determine the correlation score.

[0072] Among them, correlation detection is used to assess whether the data in the data have logical consistency and to ensure that there is an appropriate correlation between the data.

[0073] In one possible implementation, S202 specifically includes:

[0074] S2021, Based on the data to be processed, determine the relevant data units, where the relevant data units include: databases, tables, tuples and columns.

[0075] A database is a container for storing multiple tables, containing all related data structures and content. A table is the data organization form in a database, consisting of columns (fields) and rows (records). A tuple is a row of data in a table, representing a complete record of an entity or event. Columns are fields in a table, representing a type of attribute, such as "name" or "date".

[0076] S2022, Determine the correlation score based on the correlation data unit.

[0077] In this invention, the relevance score of data in a task is measured by the frequency of access.

[0078] S203, perform accuracy testing on the data to be processed and determine the accuracy score.

[0079] Accuracy testing refers to verifying the correctness of data by comparing it with real-world standards or expected values.

[0080] Specifically, through accuracy testing, errors or deviations in data can be identified and corrected, ensuring that the data is consistent with the actual situation. Accuracy scoring can quantify the correctness of data, help to discover potential problems in a timely manner, and improve the reliability of data.

[0081] In one possible implementation, S203 specifically includes:

[0082] S2031, Determine the data type of the data to be processed, where the data types include: character type and numeric type.

[0083] Character data represents text data, typically composed of letters, numbers, and symbols, such as names and addresses. Numeric data represents numerical data, usually used for mathematical operations or statistical analysis, such as age and sales figures.

[0084] S2032, Determine the accuracy score based on the data type.

[0085] When the data type is character, the accuracy score is as follows:

[0086]

[0087] Where Acc represents the accuracy score of the data to be processed, acc(r k ,D(r k )) indicates r k and D(r) k The accuracy between ) and r k D(r) represents the k-th value in the data to be processed. k ) represents r k The baseline value, NED(r) k ,D(r k )) indicates that r k Convert to D(r) k The minimum number of character insertions, deletions, or replacements required, i.e., the normalized edit distance, k=1,2,…,g, where g represents the total number of values ​​in the data to be processed.

[0088] When the data type is numeric, the accuracy score is as follows:

[0089]

[0090] Here, Max represents maximization.

[0091] S204. Combine the integrity score, relevance score, and accuracy score to determine the overall data quality score.

[0092] The overall data quality score is as follows:

[0093]

[0094] Among them, DQ total The overall data quality score is represented by W1, W2, W3, and Relevancy.

[0095] It should be noted that by independently testing different dimensions of the data, the data quality can be comprehensively assessed, thereby helping to identify problems in a timely manner and take corresponding corrective measures. By combining various scores, data quality can be measured from multiple perspectives, ensuring that multi-dimensional problems in the data are effectively identified and managed, and improving the reliability of subsequent data analysis and decision-making.

[0096] S3. Based on the comprehensive data quality score, determine the anomaly level of the data to be processed. The anomaly levels include critical anomaly, general anomaly, and minor anomaly.

[0097] It should be noted that by determining the anomaly level based on the comprehensive data quality score, the severity of data problems can be accurately distinguished, and the problems that have the most serious impact on business can be resolved first, thereby improving data management efficiency and reducing resource waste.

[0098] S4: When the data to be processed is urgent abnormal data, automatic repair is triggered directly. When the data to be processed is general abnormal data, batch scheduled repair is performed. When the data to be processed is minor abnormal data, a record is generated, awaiting subsequent manual review before repair.

[0099] It should be noted that by adopting different handling methods according to the anomaly level of the data, efficient resource allocation and problem solving are achieved. Urgent abnormal data can be processed immediately to avoid business interruption, general abnormal data can be repaired in batches to improve efficiency, and minor anomalies are left for manual review to ensure that they are handled reasonably without affecting business. This can optimize the data processing flow and improve the overall system's responsiveness and processing efficiency.

[0100] In one possible implementation, the process before S5 includes:

[0101] S8 uses the Northern Eagle algorithm to optimize the convolutional neural network.

[0102] Among them, the Northern Eagle Algorithm is a natural heuristic optimization algorithm that imitates the hunting behavior of eagles. It belongs to the category of swarm intelligence algorithms. It finds the optimal solution to a problem by simulating how eagles hunt prey. It has powerful global and local search capabilities and is widely used in function optimization and machine learning tasks.

[0103] It should be noted that by using the Northern Eagle algorithm to optimize the Naive Bayes classifier, the performance of the Naive Bayes classifier can be significantly improved. The Northern Eagle algorithm simulates the hunting process of eagles in nature and has a strong global exploration capability, which helps to escape local optima and thus find the optimal parameter configuration of the Naive Bayes classifier more efficiently. It can reach the optimal state more quickly and accurately, thereby improving the overall performance and reliability.

[0104] In one possible implementation, S8 specifically includes:

[0105] S801, initialize the population, obtain the initial population, and determine the maximum number of iterations.

[0106] S802, initialize the position and velocity of all Northern Goshawk individuals, where the position of the Northern Goshawk individual represents the parameter configuration of the Naive Bayes classifier.

[0107] S803 uses the loss function of a convolutional neural network as the fitness function to calculate the fitness value of each individual Northern Goshawk.

[0108] Specifically, the loss function is the cross-entropy loss.

[0109] S804, Update the positions of each individual Northern Goshawk based on fitness values:

[0110]

[0111]

[0112] in, This represents the position of the p-th Northern Goshawk individual in the q-th dimension after the update, x p,q P represents the position of the p-th Northern Goshawk individual in the q-th dimension. p,q This represents the position of the prey of the p-th Northern Goshawk individual in the q-th dimension, E represents a random integer with a value of 1 or 2, ω represents the inertia weight, and F... p F represents the fitness value of the p-th individual Northern Goshawk. Hp ω represents the fitness value of the prey of the p-th individual Northern Goshawk. max ω represents the maximum inertial weight value. min denoted by , where represents the minimum inertia weight value, cos represents the cosine function, d represents the current iteration number, and D represents the maximum iteration number.

[0113] Specifically, if F Hp <F p This indicates that the prey's fitness value is higher than the current Northern Goshawk's fitness value, meaning the current Northern Goshawk is attracted to the prey and will therefore move towards the prey's location and update accordingly. If F Hp ≥F p This indicates that the current Northern Goshawk individual has a good fitness value, therefore the individual will move away from the prey location and continue to search for new areas.

[0114] S805, calculates the fitness value of each individual Northern Goshawk after update.

[0115] S806: When the fitness value of the updated Northern Goshawk individual is greater than or equal to the fitness value of the current Northern Goshawk individual, update the position of the current Northern Goshawk individual. When the fitness value of the updated Northern Goshawk individual is less than the fitness value of the current Northern Goshawk individual, keep the position of the current Northern Goshawk individual unchanged.

[0116] S807, Repeat steps S802 to S806 until the maximum number of iterations is reached, and determine the optimal parameter configuration of the Naive Bayes classifier.

[0117] S5: When repairing the data to be processed, the repair strategy for the data to be processed is determined by the Naive Bayes classifier according to the data repair rules pre-configured by the system.

[0118] Data repair rules (CFD, Conditional Functional Dependency) define certain attributes of data that should meet specific values ​​under specific conditions. They typically include a premise part (LHS) and a conclusion part (RHS). Repair rules guide data repair operations to ensure data conforms to predetermined specifications. The Naive Bayes classifier is a machine learning algorithm based on Bayes' theorem. Assuming conditional independence between features, it calculates posterior probabilities based on prior and conditional probabilities to predict the category of the data to be processed or to determine the repair strategy. A repair strategy is a method determined by the system to repair data that violates rules. By analyzing data features, the system selects different repair strategies (such as RHS repair and LHS repair) to correct errors or inconsistencies in the data.

[0119] It should be noted that by using the Naive Bayes classifier, the most suitable repair strategy can be automatically selected, which improves the intelligence and automation level of data repair, increases the efficiency of the repair process, ensures the accuracy of the repair strategy, reduces manual intervention, avoids human error, and is highly adaptable, allowing for flexible application in different data scenarios.

[0120] In one possible implementation, the data repair rules are specifically as follows:

[0121]

[0122] Where CFD represents data repair rules, A i c represents the i-th attribute in the data t to be processed. i Represents attribute A i The corresponding specific constants, i=1,2,…,N, where N represents the total number of attributes in the data t to be processed, and A c This indicates the portion of the data that does not conform to the data repair rules, c c Representing constant values, A1=c1, A2=c2,…,A N =c N This is a prerequisite for data repair rules, indicating that under specific conditions, the values ​​of certain attributes should be specific constants, A. c =c c This is the conclusion part of the data repair rule, indicating that when the preconditions are met, attribute A... c The value should be equal to the constant c. c .

[0123] For example, the rule is: if ZIP=85001, then STATE=AZ.

[0124] CFD: ZIP=85001→STATE=AZ

[0125] Prerequisites (LHS): ZIP=85001

[0126] Conclusion (RHS): STATE=AZ

[0127] Data repair rule: If the ZIP of a tuple in the database is 85001, but the STATE is not AZ, then the STATE needs to be repaired to AZ.

[0128] In one possible implementation, the repair strategy specifically includes: an RHS repair strategy and an LHS repair strategy.

[0129] In this context, RHS stands for the Right-Hand Side, which is the conclusion part of Conditional Functional Dependency (CFD). The core idea of ​​the RHS repair strategy is to correct field values ​​in the data when they violate the conclusion part of the rule, ensuring that they conform to the values ​​defined in the conclusion part of the rule. LHS stands for the Left-Hand Side, which is the premise part of Conditional Functional Dependency (CFD). The core idea of ​​the LHS repair strategy is to correct field values ​​in the data when they violate the premise part of the rule, ensuring that they conform to the values ​​defined in the premise part of the rule.

[0130] In one possible implementation, S5 specifically includes:

[0131] S501, determine the prior probability and conditional probability of each repair strategy.

[0132] The prior probabilities are as follows:

[0133]

[0134] Among them, Pr(a v ) indicates the repair strategy a v The prior probability, a v Let v represent the v-th repair strategy, count() represent the number of times it occurs, M represent the total number of data points, and K represent the number of repair strategy categories.

[0135] The conditional probability is as follows:

[0136] Pr(t[ A i ]| a v )= count (t[ A i ] ∧ a v )+1 count ( a v )+K

[0137] Wherein, Pr(t[A i ]|a v ) represents a given repair strategy a v Attribute A in data t i The conditional probability of taking a specific value, [A] i ] represents attribute A in the data t to be processed. i The value of .

[0138] S502, calculate the posterior probability of each repair strategy based on the prior probability and conditional probability.

[0139] The posterior probability is as follows:

[0140] Pr( a v |t) ∝ Pr( a v ) ∏ i=1 n Pr(t[ A i ]| a v )

[0141] Among them, Pr(a v |t) represents the repair strategy a in data t. v The posterior probability is given by ∏, where ∏ represents the product operator.

[0142] S503, determine the data repair strategy based on the posterior probability.

[0143] In one possible implementation, S503 specifically includes:

[0144] When the posterior probability of the RHS repair strategy is higher than that of the LHS repair strategy, the repair strategy is the RHS repair strategy.

[0145] When the posterior probability of the RHS repair strategy is lower than that of the LHS repair strategy, the repair strategy is the LHS repair strategy.

[0146] S6 employs a repair strategy to repair the data to be processed.

[0147] It should be noted that using repair strategies to repair the data to be processed can automatically correct errors and inconsistencies in the data. By using different repair strategies (such as RHS repair and LHS repair), the repair method can be flexibly selected according to the specific problems of the data, thereby improving the accuracy and efficiency of data repair, reducing manual intervention, avoiding human error, and improving the controllability and reliability of data quality management.

[0148] The specific steps of the RHS remediation strategy are as follows:

[0149] t ' [ A c ]= c c ， foreach A i ∈ RHS

[0150] in, Indicates the repaired data, A c This indicates the portion of the data that does not conform to the data repair rules appearing in the conclusion section of the rule; c c This represents a constant value in the conclusion of the rule.

[0151] The LHS repair strategy specifically addresses the following:

[0152] t ' [ A i ]= c i ， foreach A i ∈ LHS

[0153] Among them, A i This indicates the portion of the data that does not conform to the data repair rules in the rule's premise section, c. i This represents a constant value in the premise of the rule.

[0154] For example, suppose there is a rule: Volume = 25 → Year = 2016

[0155] Data 1: Volume=25, Year=2015

[0156] The posterior probability is calculated using the Naive Bayes classifier. Assuming Pr(RHS repair|𝑡1) = 0.7 and Pr(LHS repair|𝑡1) = 0.3, the higher posterior probability indicates RHS repair.

[0157] Result: Select RHS repair, update Year to 2016.

[0158] Data 𝑡2: Volume=24, Year=2016

[0159] The posterior probability is calculated using the Naive Bayes classifier. Assuming Pr(RHS repair|𝑡1) = 0.4 and Pr(LHS repair|𝑡1) = 0.6, the higher posterior probability indicates LHS repair.

[0160] Result: Select LHS repair and change the Volume to 25.

[0161] S7: Determine if the cleanliness of the repaired data to be processed is greater than the preset cleanliness. If yes, complete the data repair. Otherwise, issue an alarm and wait for manual repair.

[0162] Data cleanliness is a key indicator for measuring the quality of repaired data. It is typically calculated by comparing the repaired data with predefined quality standards (such as completeness, accuracy, and relevance). Cleanliness can be specifically defined as the ratio of data that conforms to the repair rules to the total data.

[0163] It should be noted that by calculating the cleanliness of the repaired data and comparing it with the preset cleanliness level, the system ensures that the data repair effect meets the expected standard. It automatically determines whether further repair is needed, reducing the frequency and errors of manual intervention and improving the efficiency and accuracy of data repair. By setting the cleanliness standard, the system ensures that data quality is effectively controlled and prevents substandard data from flowing into business processes.

[0164] The beneficial effects of this invention are as follows:

[0165] In this embodiment of the invention, by calculating a comprehensive data quality score, data is categorized into urgent anomalies, general anomalies, and minor anomalies. Different repair methods are adopted according to the anomaly level, effectively improving the accuracy of data quality assessment, reducing resource waste, and enhancing the targeting and effectiveness of data repair. The repair strategy is automatically predicted through the data quality score, thereby achieving automated data repair and avoiding the errors and inefficiencies that may occur during manual repair. This significantly improves repair efficiency. A Naive Bayes classifier is used to calculate the posterior probability of each repair strategy, dynamically selecting the optimal strategy, making the repair process more intelligent. Optimal repair is performed based on the context of different data. By calculating data cleanliness and comparing it with a preset cleanliness threshold, the repair effect can be dynamically judged, ensuring that the data ultimately meets the expected quality standards. This invention, through machine learning technology and automated repair strategies, reduces the need for manual intervention, ensuring a more efficient and accurate data repair process.

[0166] System Implementation Examples

[0167] Reference manual attached Figure 2 The diagram shows a schematic representation of a machine learning-based data quality management system provided in an embodiment of the present invention.

[0168] The present invention proposes a data quality management system 30 based on machine learning, comprising: a memory 303 and a processor 301.

[0169] The memory 303 stores an application program adapted to be executed by the processor 301 to implement the big data-based oil equipment fault prediction method of the method embodiment.

[0170] The big data-based oil equipment fault prediction system 30 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302.

[0171] The structure of this machine learning-based data quality management system 30 does not constitute a limitation on the embodiments of the present invention.

[0172] Processor 301 may be a CPU, a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in connection with this disclosure. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0173] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI bus or an EISA bus, etc. Bus 302 may be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the figure, but this does not mean that there is only one bus or one type of bus.

[0174] The memory 303 may be a ROM or other type of static storage device capable of storing static information and instructions, RAM or other type of dynamic storage device capable of storing information and instructions, or it may be an EEPROM, CD-ROM or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0175] Computer-readable storage medium embodiments

[0176] The present invention proposes a computer-readable storage medium having a computer program stored thereon, the computer program being loadable and executed by a processor for a machine learning-based data quality management method of the first aspect.

[0177] The present invention has been described in detail with reference to the accompanying drawings. However, those skilled in the art should understand that the above embodiments are only preferred embodiments of the present invention. The detailed description is only to help readers better understand the spirit of the present invention, and is not intended to limit the scope of protection of the present invention. On the contrary, any improvement or modification made based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.

Claims

1. A data quality management method based on machine learning, characterized in that, The machine learning-based data quality management method includes: S1, Obtain the data to be processed; S2, perform data quality detection on the data to be processed, and determine the comprehensive data quality score of the data to be processed; S3. Based on the comprehensive data quality score, determine the anomaly level of the data to be processed, wherein the anomaly level includes critical anomaly, general anomaly, and minor anomaly. S4, when the data to be processed is urgent abnormal data, automatic repair is triggered directly; when the data to be processed is general abnormal data, batch timed repair is performed; when the data to be processed is minor abnormal data, a record is generated, and repair is performed after subsequent manual review. S5. When repairing the data to be processed, the repair strategy for the data to be processed is determined by using a Naive Bayes classifier according to the data repair rules pre-configured by the system. Specifically, S5 includes: S501, determine the prior probability and conditional probability of each of the repair strategies; S502, calculate the posterior probability of each of the repair strategies based on the prior probability and the conditional probability; S503, determine the repair strategy for the data to be processed based on the posterior probability; S6, The repair strategy is used to repair the data to be processed; S7, determine whether the cleanliness of the data to be processed after repair is greater than the preset cleanliness; if yes, complete the data repair; otherwise, issue an alarm and wait for manual repair.

2. The data quality management method based on machine learning according to claim 1, characterized in that, S2 specifically includes: S201, Perform integrity detection on the data to be processed and determine the integrity score; S202, perform correlation detection on the data to be processed and determine the correlation score; S203, perform accuracy detection on the data to be processed and determine the accuracy score; S204, combining the integrity score, the relevance score, and the accuracy score, determine the comprehensive data quality score of the data to be processed.

3. The data quality management method based on machine learning according to claim 2, characterized in that, S201 specifically includes: S2011, Based on the data to be processed, determine the integrity data unit, wherein the integrity data unit includes: column, tuple and cell; S2012, Determine the integrity score based on the integrity data unit.

4. The data quality management method based on machine learning according to claim 2, characterized in that, S202 specifically includes: S2021, Based on the data to be processed, determine the relevant data units, wherein the relevant data units include: databases, tables, tuples, and columns; S2022, Determine the correlation score based on the correlation data unit.

5. The data quality management method based on machine learning according to claim 2, characterized in that, S203 specifically includes: S2031, determine the data type of the data to be processed, wherein the data type includes: character type and numeric type; S2032, Determine the accuracy score based on the data type.

6. The data quality management method based on machine learning according to claim 1, characterized in that, The repair strategies specifically include: RHS repair strategy and LHS repair strategy.

7. The data quality management method based on machine learning according to claim 1, characterized in that, S503 specifically includes: When the posterior probability of the RHS repair strategy is higher than that of the LHS repair strategy, the repair strategy is the RHS repair strategy. When the posterior probability of the RHS repair strategy is lower than the posterior probability of the LHS repair strategy, the repair strategy is the LHS repair strategy.

8. A data quality management system based on machine learning, characterized in that, include: Memory and processor; The memory stores an application program adapted to be executed by the processor to implement the machine learning-based data quality management method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the machine learning-based data quality management method as described in any one of claims 1 to 7.

Images