Data cleaning method, device, equipment, medium and product

Abstract

The invention provides a data cleaning method, device and equipment, a medium and a product, and relates to the field of big data. Comprising the steps of obtaining to-be-cleaned data; according to a preset reinforcement learning model, a data cleaning rule is determined, the state space of the preset reinforcement learning model is the quality characteristic of the current data, the action space of the preset reinforcement learning model is cleaning operation, and the reward function of the preset reinforcement learning model is the lifting measurement based on the cleaning effect; each cleaning operation comprises at least one corresponding data cleaning rule; according to the data cleaning rule, performing data cleaning processing on the to-be-cleaned data to obtain cleaned data; determining a cleaning effect of the to-be-cleaned data according to the cleaned data and the to-be-cleaned data; and optimizing a preset reinforcement learning model according to the cleaning effect of the to-be-cleaned data. According to the method, the data cleaning flexibility and the data cleaning effect are improved.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a data cleaning method, device, equipment, medium and product. Background Art

[0002] In the process of enterprise operation, a large amount of data will be generated; during the collection, transmission, storage and processing of these large amounts of data, there may be problems such as irregular format, duplication, inconsistency, and outliers, which will affect the quality of the data and reduce the accuracy and availability of the results of analysis using these data; therefore, data cleaning is required to improve data quality.

[0003] In the prior art, predefined rules or logic are usually used to clean data to improve data quality; however, using predefined rules or logic to clean data cannot flexibly respond to different data scenarios, resulting in poor data cleaning effects and failure to meet user needs.

[0004] Based on this, in the prior art, there is a problem of insufficient flexibility in data cleaning, which leads to poor data cleaning effect. Summary of the invention

[0005] The present application provides a data cleaning method, device, equipment, medium and product to solve the technical problem of poor data cleaning effect.

[0006] In a first aspect, the present application provides a data cleaning method, comprising:

[0007] Get the data to be cleaned;

[0008] Determine a data cleaning rule according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each of the cleaning operations includes at least one corresponding data cleaning rule;

[0009] According to the data cleaning rule, the data to be cleaned is subjected to data cleaning processing to obtain cleaned data;

[0010] Determining a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned;

[0011] The preset reinforcement learning model is optimized according to the cleaning effect of the data to be cleaned.

[0012] In a second aspect, the present application provides a data cleaning device, comprising:

[0013] An acquisition module is used to acquire data to be cleaned;

[0014] A first determination module is used to determine a data cleaning rule according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each of the cleaning operations includes at least one corresponding data cleaning rule;

[0015] A cleaning processing module, used for performing data cleaning processing on the data to be cleaned according to the data cleaning rules to obtain cleaned data;

[0016] A second determination module, configured to determine a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned;

[0017] The optimization processing module is used to optimize the preset reinforcement learning model according to the cleaning effect of the data to be cleaned.

[0018] In a third aspect, the present application proposes a data cleaning device, comprising:

[0019] A processor, and a memory communicatively connected to the processor;

[0020] The memory stores computer-executable instructions;

[0021] The processor executes the computer-executable instructions stored in the memory to implement the first aspect or various possible implementations of the first aspect as described above.

[0022] In a fourth aspect, the present application proposes a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above first aspect or various possible implementation methods of the first aspect.

[0023] In a fifth aspect, the present application proposes a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect or various possible implementations of the first aspect.

[0024] The present application provides a data cleaning method, device, equipment, medium and product. The data cleaning rules are determined by a preset reinforcement learning model; the acquired data to be cleaned is subjected to data cleaning processing to obtain the cleaned data; the cleaning effect of the data to be cleaned is determined by comparing the cleaned data with the data to be cleaned; the preset reinforcement learning model is optimized according to the cleaning effect of the data to be cleaned; compared with the prior art, which uses predefined rules or logic to clean data, the present application compares the cleaned data with the data to be cleaned to obtain the cleaning effect of the data to be cleaned, and continuously optimizes the preset reinforcement learning model, thereby realizing the update of the data cleaning rules to meet the data cleaning needs of different types or different fields, and improving the flexibility and accuracy of data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] Figure 1 A data cleaning system architecture diagram provided for this application;

[0027] Figure 2 A schematic diagram of a data cleaning method provided for this application Figure 1 ;

[0028] Figure 3 A schematic diagram of a data cleaning method provided for this application Figure 2 ;

[0029] Figure 4 A schematic diagram of a data cleaning method provided for this application Figure 3 ;

[0030] Figure 5 A schematic diagram of the structure of a data cleaning device provided in this application;

[0031] Figure 6 A schematic diagram of the structure of a data cleaning device provided in this application.

[0032] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0033] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0034] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0035] In addition, this application involves conducting big data analysis of user information (including but not limited to personal biometrics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and using artificial intelligence technology to make automated decisions, and making technical solutions that have a significant impact on personal rights and interests based on the results of automated decisions. The application provides users with corresponding operation entrances for them to choose to agree or reject the results of automated decisions; if the user chooses to reject, the expert decision-making process will be entered.

[0036] It should be noted that the data cleaning method, device, equipment, medium and product provided in the present application can be used in the field of big data, and can also be used in any field other than big data. The application field of the data cleaning method, device, equipment, medium and product in the present application is not limited.

[0037] For example, a specific application scenario of the present application is to obtain a large amount of data from enterprise operations, and then use the data cleaning method of the present application to clean the acquired data to improve the quality of the data.

[0038] High-quality data is the basis for accurate analysis and decision-making. However, a large amount of data is generated in the process of enterprise operation; there are problems such as irregular format, duplication, inconsistency, and outliers. Therefore, it is necessary to clean the large amount of data obtained to improve the quality of the data and ensure the accuracy of data analysis.

[0039] In the prior art, predefined rules or logic are usually used to clean data; however, when this data cleaning method is applied to data in different fields with large differences in characteristics, it lacks a corresponding feedback mechanism and cannot adjust the predefined rules or logic accordingly, resulting in poor data cleaning effect.

[0040] Based on this, in the prior art, there are problems of insufficient flexibility in data cleaning and poor data cleaning effect.

[0041] In order to solve the above problems, the inventive concept of the present application is: by introducing a preset reinforcement learning model, corresponding data cleaning rules are formulated; and based on the data cleaning rules, data cleaning processing is performed on the acquired data to be cleaned to obtain cleaned data; by comparing the cleaned data with the data to be cleaned, the cleaning effect of the data to be cleaned is determined, and the preset reinforcement learning model is continuously optimized so that the optimized preset reinforcement learning model can update the data cleaning rules that meet the data cleaning requirements, thereby improving the flexibility and accuracy of data cleaning.

[0042] Optional, Figure 1 A schematic diagram of a data cleaning system architecture provided in an embodiment of the present application. Figure 1 In the above framework, the above framework includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0043] It is understandable that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the data cleaning system architecture. In other feasible implementations of the present application, the above architecture may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0044] In a specific implementation process, the data acquisition device 101 may include an input / output interface and may also include a communication interface. The data acquisition device 101 may be connected to the processing device via the input / output interface or the communication interface.

[0045] The processing device 102 can determine the data cleaning rules according to the preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is the improvement metric based on the cleaning effect, and each cleaning operation includes at least one corresponding data cleaning rule; according to the data cleaning rules, the data to be cleaned is subjected to data cleaning processing to obtain cleaned data; according to the cleaned data and the data to be cleaned, the cleaning effect of the data to be cleaned is determined; according to the cleaning effect of the data to be cleaned, the preset reinforcement learning model is optimized.

[0046] The display device 103 may also be a touch display screen or a screen of a terminal device, which is used to receive user instructions while displaying the above content to achieve interaction with the user.

[0047] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0048] Figure 2 A schematic diagram of a data cleaning method provided for this application Figure 1 ,like Figure 2 As shown, the method includes:

[0049] S201, obtaining data to be cleaned.

[0050] Optionally, obtain the data to be cleaned, including:

[0051] Data to be cleaned is received from multiple data sources, wherein the data sources include multiple data external to the enterprise, procurement data, and internal data of the enterprise.

[0052] In this embodiment, by receiving the data to be cleaned from multiple data sources, the cleaning requirements of the data to be cleaned from different data sources are clarified, so as to improve the processing efficiency of the data to be cleaned from different data sources;

[0053] For example, the main problems that may exist with an enterprise's external data are inconsistent formats and a large number of missing values; the main problems that may exist with procurement data are data inconsistency and data duplication; the main problem that may exist with an enterprise's internal data is data redundancy; for data from different data sources, the use of reasonable and corresponding data cleaning rules can improve the processing efficiency of data cleaning.

[0054] S202. Determine data cleaning rules according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each cleaning operation includes at least one corresponding data cleaning rule.

[0055] In this embodiment, the quality characteristics of current data include accuracy, completeness, consistency, and processing efficiency;

[0056] Optionally, the cleaning operation includes data preprocessing, desensitization processing, outlier detection and processing, deduplication processing and missing value filling processing.

[0057] In this embodiment, sensitive data includes identity card number, bank account number and mobile phone number; the sensitive data in the data to be cleaned is automatically identified, and the automatically identified sensitive data is encrypted or displayed in a blurred form to improve the security of the data to be cleaned.

[0058] In this embodiment, the availability of the data to be cleaned is improved by preprocessing the data to be cleaned;

[0059] By detecting and processing outliers in the data to be cleaned, the accuracy of the data to be cleaned is improved;

[0060] By deduplicating the data to be cleaned, the amount of redundant data to be cleaned is reduced, ensuring the uniqueness of the data to be cleaned;

[0061] By filling in missing values ​​in the data to be cleaned, the integrity of the data to be cleaned is improved.

[0062] S203: Perform data cleaning processing on the data to be cleaned according to the data cleaning rules to obtain cleaned data.

[0063] In this embodiment, different data cleaning rules are used for different problems in the data to be cleaned, and targeted data cleaning processing is performed on the data to be cleaned, thereby improving the flexibility of data cleaning and the data cleaning effect.

[0064] S204: Determine the cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned.

[0065] Optionally, determining a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned includes:

[0066] According to the cleaned data and the data to be cleaned, a cleaning effect evaluation value of the data to be cleaned is determined, wherein the cleaning effect evaluation value includes an accuracy evaluation value, an integrity evaluation value, a consistency evaluation value and a processing efficiency evaluation value.

[0067] In this embodiment, the accuracy evaluation value refers to the degree of match between the cleaned data and the preset standard value; the completeness evaluation value refers to the ratio of the number of missing values ​​to the total number of data to be cleaned; the processing efficiency evaluation value refers to the number of data to be cleaned cleaned per second; and the consistency evaluation value refers to whether the time series between the data to be cleaned conforms to the time logic.

[0068] Through multi-dimensional cleaning effect evaluation values, it provides comprehensive cleaning effects of the data to be cleaned, improving the flexibility and accuracy of data cleaning.

[0069] S205: Optimize the preset reinforcement learning model according to the cleaning effect of the data to be cleaned.

[0070] Alternatively, if Figure 3 As shown in the figure, according to the cleaning effect of the data to be cleaned, the optimized preset reinforcement learning model includes:

[0071] S2051. Determine, according to the cleaning effect evaluation value, a cleaning effect-based improvement metric corresponding to the data to be cleaned.

[0072] In this embodiment, for example, if the accuracy evaluation value of the data to be cleaned is less than 95%, or the integrity evaluation value is greater than 2%, or the processing efficiency evaluation value is less than a preset processing efficiency threshold, or the consistency evaluation value is inconsistent;

[0073] Based on the corresponding cleaning effect evaluation value, the preset reinforcement learning model is used to calculate the improvement metric based on the cleaning effect corresponding to the data to be cleaned; wherein the improvement metric based on the cleaning effect corresponding to the data to be cleaned is the reward function of the preset reinforcement learning model , the calculation formula is as follows:

[0074]

[0075] Where t is time, S t is the state space at time t, a t is the cleaning operation performed at time t, Q(S t , a t ) is in the state space S t Take the cleaning action performed a t The value of is the learning rate ( greater than 0 and less than or equal to 1), is the reward function, is the discount factor ( greater than or equal to 0 and less than 1), S t+1 is the state space at time t+1; The state space S t+1The maximum value of all possible cleaning operations.

[0076] S2052. Optimize the preset reinforcement learning model according to the improvement metric based on the cleaning effect corresponding to the data to be cleaned.

[0077] In this embodiment, according to the improvement metric based on the cleaning effect corresponding to the data to be cleaned, the accuracy assessment threshold, the integrity assessment threshold, the preset processing efficiency threshold or the consistency assessment value of the data to be cleaned are adjusted to achieve the optimization of the preset reinforcement learning model;

[0078] Furthermore, the data cleaning rules are updated through the optimized preset reinforcement learning model, and the updated data cleaning rules are applied to the data cleaning process to improve the flexibility and accuracy of data cleaning.

[0079] The data cleaning method proposed in this application automatically generates data cleaning rules through a preset reinforcement learning model, and combines the data cleaning effect with the feedback mechanism to achieve automatic adjustment of the data cleaning rules to adapt to the cleaning needs of the data to be cleaned from different data sources, thereby improving the flexibility and accuracy of the data.

[0080] Figure 4 A schematic diagram of a data cleaning method provided for this application Figure 3 ,exist Figure 2 Based on the embodiment shown, Figure 4 As shown, in the above step S203, the data to be cleaned is cleaned according to the data cleaning rule, including:

[0081] S401. According to the data cleaning rules corresponding to the data preprocessing, perform at least one of noise removal, formatting processing and language processing on the data to be cleaned.

[0082] In this embodiment, noise removal is to clean up blank characters, special symbols and redundant data in the data to be cleaned by regular expressions;

[0083] Formatting is to unify the data format including date, time and currency unit in the data to be cleaned;

[0084] Language processing is to separate the mixed Chinese and English content in the data to be cleaned and perform standardized processing.

[0085] In this embodiment, the accuracy and consistency of the data to be cleaned are improved through noise removal, formatting processing and language processing, thereby improving the data quality.

[0086] S402: Detect abnormal data using statistical methods according to data cleaning rules corresponding to abnormal value detection and processing.

[0087] In this embodiment, the statistical method includes the three standard deviation rule. For example, the statistical method is used to detect abnormal data, including:

[0088] Calculate the mean and standard deviation of the data to be cleaned;

[0089] Based on the characteristics of normal distribution, the data to be cleaned that are beyond the range of positive and negative three standard deviations of the mean of the data to be cleaned are regarded as abnormal data.

[0090] In this embodiment, the statistical method of the three-sigma rule is used to achieve rapid detection of abnormal data in the data to be cleaned, thereby improving the efficiency of identifying abnormal data.

[0091] S403: Perform outlier processing on the abnormal data by combining the clustering algorithm and the context information.

[0092] In this embodiment, the clustering algorithm includes a K-means clustering algorithm, a density-based clustering algorithm, a hierarchical clustering algorithm or a Gaussian mixture model; the abnormal data is further detected by different clustering algorithms, thereby improving the flexibility and accuracy of abnormal data detection.

[0093] Furthermore, the rationality of abnormal data is judged through context information, and the outliers are corrected or deleted through interpolation or regression analysis to improve the accuracy of abnormal data.

[0094] S404: According to the data cleaning rule corresponding to the deduplication process, the hash value of the data to be cleaned is used to locate the duplicate records.

[0095] In this embodiment, a corresponding unique hash value is generated for each data to be cleaned, and the unique hash value is stored in a hash table; before inserting the unique hash value of the next data to be cleaned into the hash table, a comparison is made to see whether the unique hash value already exists in the hash table. If the unique hash value already exists, it indicates that the data to be cleaned is a duplicate record.

[0096] In this embodiment, the hash algorithm is suitable for calculation and comparison of large-scale data sets. The hash algorithm is used to preliminarily detect duplicate records in the data to be cleaned, thereby improving the speed of deduplication detection.

[0097] S405: According to the data cleaning rules corresponding to the deduplication process, a text similarity calculation method is used to identify and merge duplicate records.

[0098] In this embodiment, further, a text similarity calculation method is used to identify duplicate records to improve the accuracy of deduplication detection. If the similarity of the duplicate records is greater than a preset similarity threshold, the duplicate records are merged.

[0099] S406: Determine the first type of missing data and the second type of missing data in the data to be cleaned according to the data cleaning rules corresponding to the missing value filling.

[0100] In this embodiment, the missing value ratio of the data to be cleaned is calculated. If the missing value ratio of the data to be cleaned is less than the preset missing value threshold, the missing value is determined as the first type of missing data in the data to be cleaned; if the missing value ratio of the data to be cleaned is greater than or equal to the preset missing value threshold, the missing value is determined as the second type of missing data in the data to be cleaned.

[0101] S407. Use mean or median filling method to perform data cleaning on the first type of missing data.

[0102] In this embodiment, for the first type of missing data, a mean or median filling method is used to perform data cleaning processing on the first type of missing data, which occupies less computing resources and avoids waste of computing resources.

[0103] S408. Use machine learning to perform data cleaning on the second type of missing data.

[0104] In this embodiment, the machine learning method includes random forest or generative adversarial network; through the machine learning method, the second type of missing data is cleaned and processed, the missing value filling is quickly realized, and the efficiency of missing value filling is improved.

[0105] In this embodiment, by classifying the missing data, different missing value filling processing methods are adopted, thereby avoiding the waste of computing resources or the low efficiency of missing value filling.

[0106] Optionally, the above step S401 is example one, steps S402 to S403 are example two, steps S404 to S405 are example three, and steps S406 to S408 are example four; then in the above step S204, data cleaning processing is performed on the data to be cleaned according to the data cleaning rule, including any one, any two, any three or a combination of all examples in example one, example two, example three and example four.

[0107] Also optionally, the data cleaning method proposed in the present application, after acquiring the data to be cleaned, decomposes the data to be cleaned uploaded to the distributed platform through a distributed platform (Apache Spark distributed computing framework) based on MapReduce (a programming model and framework for processing large-scale data sets) technology to obtain independent data blocks, and assigns them to multiple reduction tasks to achieve parallel data cleaning processing of independent data blocks, thereby improving the efficiency of data cleaning processing.

[0108] Figure 5A schematic diagram of the structure of a data cleaning device provided in this application, such as Figure 5 As shown, the data cleaning device includes:

[0109] An acquisition module 501 is used to acquire data to be cleaned;

[0110] A first determination module 502 is used to determine a data cleaning rule according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each cleaning operation includes at least one corresponding data cleaning rule;

[0111] Optionally, the cleaning operation includes data preprocessing, desensitization processing, outlier detection and processing, deduplication processing and missing value filling processing.

[0112] The cleaning processing module 503 is used to perform data cleaning processing on the data to be cleaned according to the data cleaning rules to obtain cleaned data;

[0113] A second determination module 504 is used to determine the cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned;

[0114] The optimization processing module 505 is used to optimize the preset reinforcement learning model according to the cleaning effect of the data to be cleaned.

[0115] Optionally, the acquisition module 501 is further configured to:

[0116] Data to be cleaned is received from multiple data sources, wherein the data sources include multiple data external to the enterprise, procurement data, and internal data of the enterprise.

[0117] Optionally, the cleaning processing module 503 is further used for:

[0118] According to the data cleaning rules corresponding to the data preprocessing, at least one of noise removal, formatting processing and language processing is performed on the data to be cleaned.

[0119] Optionally, the cleaning processing module 503 is further used for:

[0120] According to the data cleaning rules corresponding to outlier detection and processing, statistical methods are used to detect abnormal data;

[0121] Clustering algorithms and context information are combined to perform outlier processing on abnormal data.

[0122] Optionally, the cleaning processing module 503 is further used for:

[0123] According to the data cleaning rules corresponding to the deduplication process, the hash value of the data to be cleaned is used to locate duplicate records;

[0124] According to the data cleaning rules corresponding to the deduplication process, the text similarity calculation method is used to identify and merge duplicate records.

[0125] Optionally, the cleaning processing module 503 is further used for:

[0126] According to the data cleaning rules corresponding to the missing value filling, the first type of missing data and the second type of missing data in the data to be cleaned are determined;

[0127] The first type of missing data is cleaned by filling in the mean or median;

[0128] Machine learning is used to clean the second type of missing data.

[0129] Optionally, the second determining module 504 is further configured to:

[0130] According to the cleaned data and the data to be cleaned, a cleaning effect evaluation value of the data to be cleaned is determined, wherein the cleaning effect evaluation value includes an accuracy evaluation value, an integrity evaluation value, a consistency evaluation value and a processing efficiency evaluation value.

[0131] Optionally, the optimization processing module 505 is further configured to:

[0132] According to the cleaning effect evaluation value, determine the improvement metric based on the cleaning effect corresponding to the data to be cleaned;

[0133] Optimize the preset reinforcement learning model according to the improvement metric based on the cleaning effect corresponding to the data to be cleaned.

[0134] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0135] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0136] Figure 6 A structural diagram of a data cleaning device provided in this application, such as Figure 6 As shown, the data cleaning device includes: at least one processor 601 and a memory 602. Optionally, the device also includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected via a bus 604.

[0137] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above method.

[0138] The specific implementation process of the processor 601 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0139] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the invention can be directly implemented as a hardware processor, or can be implemented by a combination of hardware and software modules in the processor.

[0140] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0141] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0142] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0143] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0144] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0145] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist in a device as discrete components.

[0146] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0149] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes.

[0150] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0151] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data cleaning method, characterized in that: include: Get the data to be cleaned; Determine a data cleaning rule according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each of the cleaning operations includes at least one corresponding data cleaning rule; According to the data cleaning rule, the data to be cleaned is subjected to data cleaning processing to obtain cleaned data; Determining a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned; The preset reinforcement learning model is optimized according to the cleaning effect of the data to be cleaned.

2. The method according to claim 1, characterized in that: The step of determining a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned includes: A cleaning effect evaluation value of the data to be cleaned is determined according to the cleaned data and the data to be cleaned, wherein the cleaning effect evaluation value includes an accuracy evaluation value, an integrity evaluation value, a consistency evaluation value and a processing efficiency evaluation value.

3. The method according to claim 2, characterized in that The optimizing the preset reinforcement learning model according to the cleaning effect of the data to be cleaned includes: Determining, according to the cleaning effect evaluation value, a cleaning effect-based improvement metric corresponding to the to-be-cleaned data; The preset reinforcement learning model is optimized according to the improvement metric based on the cleaning effect corresponding to the data to be cleaned.

4. The method according to claim 1, characterized in that: The step of obtaining the data to be cleaned includes: Data to be cleaned is received from multiple data sources, wherein the data sources include multiple data from external enterprises, procurement data, and internal enterprise data.

5. The method according to any one of claims 1 to 3, characterized in that: The cleaning operation includes data preprocessing, desensitization processing, outlier detection and processing, deduplication processing and missing value filling processing.

6. The method according to claim 5, characterized in that The step of performing data cleaning processing on the data to be cleaned according to the data cleaning rule includes: According to the data cleaning rules corresponding to the data preprocessing, at least one of noise removal, formatting processing and language processing is performed on the data to be cleaned.

7. The method according to claim 5, characterized in that The step of performing data cleaning processing on the data to be cleaned according to the data cleaning rule includes: According to the data cleaning rules corresponding to the outlier detection and processing, a statistical method is used to detect abnormal data; The abnormal data is processed by combining a clustering algorithm and context information.

8. The method according to claim 5, characterized in that The step of performing data cleaning processing on the data to be cleaned according to the data cleaning rule includes: According to the data cleaning rule corresponding to the deduplication processing, the hash value of the data to be cleaned is used to locate the duplicate records; According to the data cleaning rules corresponding to the deduplication process, a text similarity calculation method is adopted to identify and merge the duplicate records.

9. The method according to claim 5, characterized in that The step of performing data cleaning processing on the data to be cleaned according to the data cleaning rule includes: Determine the first type of missing data and the second type of missing data in the data to be cleaned according to the data cleaning rules corresponding to the missing value filling; Using mean or median filling method to clean the first type of missing data; The second type of missing data is cleaned using machine learning methods.

10. A data cleaning device, characterized in that: include: An acquisition module is used to acquire data to be cleaned; A first determination module is used to determine a data cleaning rule according to a preset reinforcement learning model, wherein the state space of the preset reinforcement learning model is the quality characteristics of the current data, the action space of the preset reinforcement learning model is the cleaning operation, the reward function of the preset reinforcement learning model is an improvement metric based on the cleaning effect, and each of the cleaning operations includes at least one corresponding data cleaning rule; A cleaning processing module, used for performing data cleaning processing on the data to be cleaned according to the data cleaning rules to obtain cleaned data; A second determination module, configured to determine a cleaning effect of the data to be cleaned according to the cleaned data and the data to be cleaned; The optimization processing module is used to optimize the preset reinforcement learning model according to the cleaning effect of the data to be cleaned.

11. A data cleaning device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.

13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.

Images