Intelligent data cleaning method and device based on big data and electronic equipment

Abstract

The invention relates to an intelligent data cleaning method and device based on big data and electronic equipment, and belongs to the technical field of big data.The method comprises the steps that multi-source service data is obtained, a source data table is established according to the multi-source service data, and a target data table with a mapping relation is established according to the source data table; determining a cleaning rule according to the cleaning requirement and the cleaning purpose; according to the type and the type of the multi-source business data, cleaning the multi-source business data in the source data table by adopting a corresponding cleaning rule, and storing the cleaned data into a target data table; and counting the use frequency of each cleaning rule, and automatically configuring the cleaning rule according to the use frequency of the cleaning rule. The technical problem that in the prior art, data are not automatically cleaned, a large amount of manpower and time cost needs to be input, and consequently the data cleaning efficiency is low is solved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device and electronic equipment for intelligent data cleaning based on big data. Background Art

[0002] With the continuous development of informatization, the application of cloud computing and big data has been highly valued by governments, scientific research institutions, and industry manufacturers. In order to improve the comprehensive utilization rate of data resources and solve the problems of "information islands" and inconsistent, non-standardized, and incomplete data in traditional business systems, all industries need to establish big data centers.

[0003] Big data centers are powerful tools for business experts to organize and analyze various types of massive business data. However, the original data standards are not consistent and cannot be used directly. Data availability must be improved through data cleaning before it can be used. The existing technology does not automatically clean the data, and a large number of people are required to manually clean the data. This will increase the manpower and time costs of users, which is time-consuming, labor-intensive and inefficient. Summary of the invention

[0004] In view of this, it is necessary to provide an intelligent data cleaning method, device and electronic device based on big data to solve the technical problem that the prior art does not automatically clean the data, requires a lot of manpower and time costs, and thus leads to low data cleaning efficiency.

[0005] In order to solve the above problems, the present invention provides an intelligent data cleaning method based on big data, comprising: Acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; Determine cleaning rules based on cleaning needs and cleaning purposes; According to the type and category of multi-source business data, use the corresponding cleaning rules to clean the multi-source business data in the source data table, and store the cleaned data in the target data table; The usage frequency of each cleaning rule is counted, and the cleaning rules are automatically configured according to the usage frequency of the cleaning rules.

[0006] In a possible implementation, the step of establishing a source data table according to multi-source business data, and establishing a target data table having a mapping relationship according to the source data table, includes: Determine the structure of the source data table based on multi-source business data, and define the multi-source business data type and category corresponding to each field in the source data table; Establish the mapping relationship between the source data table and the target data table based on table mapping, field mapping and data conversion rules.

[0007] In a possible implementation, before determining the cleaning rule, the following is further included: Classify and label multi-source business data to determine the types and categories of multi-source business data; Preprocess the marked multi-source business data to remove leading and trailing spaces, convert letter case, and convert Chinese characters from full-width to half-width.

[0008] In a possible implementation, the cleaning rule at least includes: According to the data classification tags, establish cleaning rules to delete duplicate data or store them separately; Perform preset standardized conversion on date and time format data; Perform preset standardized conversions on digital format data; Perform preset standardized conversions on different types of data according to the type dictionary table as required; Replace and decompose special symbols in strings; Extract the data that needs to be extracted from the string according to the regular expression.

[0009] In a possible implementation, the step of using corresponding cleaning rules to clean the multi-source business data in the source data table and storing the cleaned data in the target data table includes: Establish an abnormal data storage table; Use corresponding cleaning rules to clean the multi-source business data in the source data table, and store the successfully cleaned multi-source business data in the target data table; And the data that failed to be cleaned is stored in the abnormal data storage table.

[0010] In a possible implementation, counting the usage frequency of each cleaning rule and automatically configuring the cleaning rule according to the usage frequency of the cleaning rule includes: Based on each multi-source business data cleaning process, the number of times the cleaning rules involved are configured is counted; According to the statistical results, the usage frequency of each cleaning rule is sorted to determine the utilization score; Automatically configure cleaning rules based on utilization scores from high to low.

[0011] In a possible implementation, sorting the usage frequency of each cleaning rule according to the statistical result to determine the utilization score includes: The utilization score is determined based on the ratio of the number of times the cleaning rule is used to the total number of times all rules are called.

[0012] In a second aspect, the present invention further provides an intelligent data cleaning device based on big data, comprising: An acquisition module is used to acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; A rule determination module is used to determine cleaning rules according to cleaning requirements and cleaning purposes; A cleaning module is used to clean the multi-source business data in the source data table according to the type and category of the multi-source business data using corresponding cleaning rules, and store the cleaned data in the target data table; The rule configuration module is used to count the usage frequency of each cleaning rule and automatically configure the cleaning rules according to the usage frequency of the cleaning rules.

[0013] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps in the intelligent data cleaning method based on big data as described above are implemented.

[0014] In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the intelligent data cleaning method based on big data as described above.

[0015] The beneficial effects of the present invention are as follows: first, multi-source business data is acquired, a source data table is established according to the multi-source business data, and a target data table with a mapping relationship is established according to the source data table; and cleaning rules are determined according to the cleaning requirements and cleaning purposes; according to the types and categories of the multi-source business data, the multi-source business data in the source data table is cleaned using corresponding cleaning rules, and the cleaned data is stored in the target data table; the purpose of automatically cleaning data through cleaning rules is achieved. Finally, the frequency of use of each cleaning rule is counted, and the cleaning rule is automatically configured according to the frequency of use of the cleaning rule. The frequency of use is determined by iterative accumulation of cleaning standards for different types of data, so that according to the data classification of the user-defined data table and data field, the intelligent rule library automatically matches the data cleaning rules according to the data classification and frequency of use, and the user does not need to manually configure the data cleaning rules. The user can edit the automatically configured cleaning rules, which minimizes manual data cleaning and achieves the purpose of automatic data cleaning. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A method flow chart of an embodiment of an intelligent data cleaning method based on big data provided by the present invention; Figure 2A schematic diagram of an implementation method of an intelligent data cleaning method based on big data provided by the present invention; Figure 3 A method flow chart of an embodiment of step S101 in the intelligent data cleaning method based on big data provided by the present invention; Figure 4 A method flow chart of an embodiment of step S103 in the intelligent data cleaning method based on big data provided by the present invention; Figure 5 A method flow chart of an embodiment of step S104 in the intelligent data cleaning method based on big data provided by the present invention; Figure 6 is a schematic diagram of an embodiment of an intelligent data cleaning device based on big data provided by the present invention; Figure 7 It is a schematic diagram of the operating environment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0017] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0018] A specific embodiment of the present invention discloses an intelligent data cleaning method based on big data, please refer to Figure 1 and Figure 2 ,include: S101, acquiring multi-source business data, establishing a source data table according to the multi-source business data, and establishing a target data table having a mapping relationship according to the source data table; S102, determining cleaning rules according to cleaning requirements and cleaning purposes; S103, according to the type and category of the multi-source business data, use corresponding cleaning rules to clean the multi-source business data in the source data table, and store the cleaned data in the target data table; S104: Count the usage frequency of each cleaning rule, and automatically configure the cleaning rule according to the usage frequency of the cleaning rule.

[0019] In this embodiment, multi-source business data is first obtained, a source data table is established according to the multi-source business data, and a target data table with a mapping relationship is established according to the source data table; and the cleaning rules are determined according to the cleaning requirements and cleaning purposes; according to the type and category of the multi-source business data, the corresponding cleaning rules are used to clean the multi-source business data in the source data table, and the cleaned data is stored in the target data table; the purpose of automatically cleaning the data through the cleaning rules is achieved. Finally, the frequency of use of each cleaning rule is counted, and the cleaning rules are automatically configured according to the frequency of use of the cleaning rules. The frequency of use is determined by iterative accumulation of cleaning standards for different types of data, so that according to the data classification of the user-defined data table and data field, the intelligent rule library automatically matches the data cleaning rules according to the data classification and frequency of use, without the need for the user to manually configure the data cleaning rules. The user can edit the automatically configured cleaning rules, which minimizes manual data cleaning and achieves the purpose of automatic data cleaning.

[0020] In step S101, different data collection methods are used to obtain multi-source business data according to different business data sources, and the data is persisted in a local database. The scope and type of the business are not limited.

[0021] In step S102, the cleaning rule base refers to a system or module that centrally stores and manages data cleaning rules. It contains a series of predefined, reusable rules designed to clean and transform raw data to improve data quality and consistency. Before determining the cleaning rules, first conduct a comprehensive quality assessment of the source data to identify and record problems in the data. Key issues include null values, format errors, duplicate data, outliers, etc. Use statistical methods: Understand the overall situation of the data through data summary statistics, distribution analysis and other methods.

[0022] In some embodiments, the source data table is established according to the multi-source business data, and the target data table with a mapping relationship is established according to the source data table. Figure 3 ,include: S301. Determine the structure of the source data table according to the multi-source business data, and define the multi-source business data type and category corresponding to each field in the source data table; S302: Establish a mapping relationship between a source data table and a target data table according to table mapping, field mapping and data conversion rules.

[0023] In this embodiment, the source data table is used to store the original data obtained from multiple business systems or data sources. Each source data table corresponds to a specific business data source and usually contains multiple fields, each field representing an attribute of the data. According to the characteristics and requirements of the data source, the data table structure is designed, including the table name, field name, field type, etc.

[0024] The target data table is the target table that stores the processed and cleaned data, which is usually used for analysis, reporting, and decision support. The cleaned and transformed data is loaded from the source data table into the target data table.

[0025] Furthermore, establishing a mapping relationship between the source data table and the target data table includes but is not limited to: 1. Table mapping design Analyze business needs and understand business requirements, such as which reports or analysis results need to be output.

[0026] Classify data sources, determine the source business system, and classify them according to business logic, such as sales, customers, inventory, etc.

[0027] Summarize the data structure and record the structure of the source data table in detail, including table name, field name, field type, and sample data.

[0028] 2. Field mapping design One-to-one / many-to-one, determine how each field in the source data table is mapped to the field in the target data table. Sometimes one source field may correspond to multiple target fields, and vice versa.

[0029] 3. Data conversion rules According to the field mapping design, list the required conversion rules. For example, you may need to convert the date format from "DD / MM / YYYY" to "YYYY-MM-DD".

[0030] Determine the rules for data standardization, such as removing invalid characters, adjusting uppercase and lowercase, and unifying units.

[0031] In a specific embodiment, the relationship between data tables, data fields and field cleaning rules is manually defined in the intelligent rule library, and the field data classification is identified. A mapping relationship is established between the source data table and the target data table, and a mapping relationship is established between the source data table fields and the target data table fields. The cleaning rules for the stored data table fields are configured, and the field data classification is identified. For example: a mapping relationship is established between the source data table AA and the target data table BB, and a mapping relationship is established between the A1, A2, and A3 fields in the AA table and the B1, B2, and B3 fields in the BB table. The data in the A1 field is written into the B1 field, and so on. The cleaning rules for the B1 field can be configured to automatically complete the data for the B1 field. The data type of the B1 field is identified, such as date data. When the field is not configured with data cleaning rules, the system will match the default cleaning rules according to the field classification.

[0032] In some embodiments, before determining the cleaning rules, the method further includes: Classify and label multi-source business data to determine the types and categories of multi-source business data; Preprocess the marked multi-source business data to remove leading and trailing spaces, convert letter case, and convert Chinese characters from full-width to half-width.

[0033] In this embodiment, during the data preprocessing process, the business data preprocessing adaptive configuration function is used to classify and mark the business data with different parameter characteristics according to the data mapping relationship setting of the data cleaning rule library, and remove the leading and trailing spaces of the data, the upper and lower case conversion of letters, and the conversion of full-width to half-width Chinese characters. Data that fails preprocessing is stored separately.

[0034] In a specific embodiment, the data identifier record in the cleaning rule library is matched according to the data identifier (table name). If the match fails, the preprocessing is determined to have failed. If the match succeeds, field matching is performed according to the field mapping relationship of the table in the cleaning rule library. If the match fails (the success criterion is given by the cleaning rule library), the preprocessing is determined to have failed. If the match succeeds, general data processing is performed, such as removing leading and trailing spaces, and the preprocessing is completed.

[0035] Furthermore, the cleaning rules at least include: According to the data classification tags, establish cleaning rules to delete duplicate data or store them separately; Perform preset standardized conversion on date and time format data; Perform preset standardized conversions on digital format data; Perform preset standardized conversions on different types of data according to the type dictionary table as required; Replace and decompose special symbols in strings; Extract the data that needs to be extracted from the string according to the regular expression.

[0036] In some embodiments, the corresponding cleaning rules are used to clean the multi-source business data in the source data table, and the cleaned data is stored in the target data table. Figure 4 ,include: S401, establishing an abnormal data storage table; S402, using corresponding cleaning rules to clean the multi-source business data in the source data table, and storing the cleaned multi-source business data in the target data table; S403, and storing the data that failed to be cleaned in the abnormal data storage table.

[0037] In this embodiment, the successfully cleaned data is stored in a specially set abnormal data storage table. On the one hand, the accuracy of cleaning can be improved. On the other hand, the effectiveness and accuracy of data cleaning can be further improved by subsequent manual processing of the data in the abnormal data storage table.

[0038] In some embodiments, the usage frequency of each cleaning rule is counted, and the cleaning rule is automatically configured according to the usage frequency of the cleaning rule. Figure 5 ,include: S501, based on each multi-source business data cleaning process, counting the number of times the cleaning rules involved are configured; S502, sorting the usage frequency of each cleaning rule according to the statistical results to determine the utilization score; S503. Automatically configure cleaning rules from high to low according to the utilization scores.

[0039] In this embodiment, the number of times the cleaning rules are configured by users in the intelligent rule base is counted by cleaning rule category, the data cleaning rules are sorted according to the number of configurations, and the cleaning rules are given utilization scores according to the ranking. The first ranked cleaning rule is scored as 100 points, the second is scored as 99 points, and so on for all data cleaning rules. The utilization score provides a basis for judging intelligent data cleaning.

[0040] According to the data classification of user-defined data tables and data fields, the intelligent rule library automatically matches data cleaning rules according to data classification and utilization scoring standards. Users do not need to manually configure data cleaning rules. Users can edit the automatically configured cleaning rules.

[0041] It should be noted that by manually configuring the intelligent rule base, new cleaning rules or data mapping relationships can be added, and existing cleaning rules and data mapping relationships can be viewed, edited, and deleted.

[0042] Based on the above-mentioned intelligent data cleaning method based on big data, the embodiment of the present application further provides an intelligent data cleaning device 600 based on big data, including: The acquisition module 610 is used to acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; A rule determination module 620, for determining cleaning rules according to cleaning requirements and cleaning purposes; A cleaning module 630, for cleaning the multi-source business data in the source data table using corresponding cleaning rules according to the types and categories of the multi-source business data, and storing the cleaned data in the target data table; The rule configuration module 640 is used to count the usage frequency of each cleaning rule and automatically configure the cleaning rule according to the usage frequency of the cleaning rule.

[0043] like Figure 7As shown, based on the above-mentioned intelligent data cleaning method based on big data, the present invention also provides an electronic device, which can be a computing electronic device such as a mobile terminal, a desktop computer, a notebook, a palm computer, and a server. The electronic device includes a processor 710, a memory 720, and a display 730. Figure 7 Only some components of the electronic device are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0044] In some embodiments, the memory 720 may be an internal storage unit of the electronic device, such as a hard disk or memory of the electronic device. In other embodiments, the memory 720 may also be an external storage electronic device of the electronic device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 720 may also include both an internal storage unit of the electronic device and an external storage electronic device. The memory 720 is used to store application software and various types of data installed in the electronic device, such as program codes for installing the electronic device. The memory 720 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 720 stores a smart cleaning data program 740 based on big data, and the smart cleaning data program 740 based on big data can be executed by the processor 710, thereby realizing the smart cleaning data method based on big data in each embodiment of the present application.

[0045] In some embodiments, the processor 710 may be a central processing unit (CPU), a microprocessor or other data processing chip, used to run program codes stored in the memory 720 or process data, such as executing an intelligent data cleaning method based on big data.

[0046] In some embodiments, the display 730 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 730 is used to display information on the electronic device for intelligent cleaning data based on big data and to display a visual user interface. The components 710-730 of the electronic device communicate with each other via a system bus.

[0047] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0048] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An intelligent data cleaning method based on big data, characterized in that: include: Acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; Determine cleaning rules based on cleaning needs and cleaning purposes; According to the type and category of multi-source business data, use the corresponding cleaning rules to clean the multi-source business data in the source data table, and store the cleaned data in the target data table; The usage frequency of each cleaning rule is counted, and the cleaning rules are automatically configured according to the usage frequency of the cleaning rules.

2. The intelligent data cleaning method based on big data according to claim 1 is characterized in that: The step of establishing a source data table according to the multi-source business data, and establishing a target data table having a mapping relationship according to the source data table, includes: Determine the structure of the source data table based on multi-source business data, and define the multi-source business data type and category corresponding to each field in the source data table; Establish the mapping relationship between the source data table and the target data table based on table mapping, field mapping and data conversion rules.

3. The intelligent data cleaning method based on big data according to claim 1 is characterized in that: Before determining the cleaning rules, it also includes: Classify and label multi-source business data to determine the types and categories of multi-source business data; Preprocess the marked multi-source business data to remove leading and trailing spaces, convert letter case, and convert Chinese characters from full-width to half-width.

4. The intelligent data cleaning method based on big data according to claim 3 is characterized in that: The cleaning rules at least include: According to the data classification tags, establish cleaning rules to delete duplicate data or store them separately; Perform preset standardized conversion on date and time format data; Perform preset standardized conversions on digital format data; Perform preset standardized conversions on different types of data according to the type dictionary table as required; Replace and decompose special symbols in strings; Extract the data that needs to be extracted from the string according to the regular expression.

5. The intelligent data cleaning method based on big data according to claim 4 is characterized in that: The method of using corresponding cleaning rules to clean the multi-source business data in the source data table and storing the cleaned data in the target data table includes: Establish an abnormal data storage table; Use corresponding cleaning rules to clean the multi-source business data in the source data table, and store the successfully cleaned multi-source business data in the target data table; And the data that failed to be cleaned is stored in the abnormal data storage table.

6. The intelligent data cleaning method based on big data according to claim 1 is characterized in that: The counting of the usage frequency of each cleaning rule and automatically configuring the cleaning rule according to the usage frequency of the cleaning rule include: Based on each multi-source business data cleaning process, the number of times the cleaning rules involved are configured is counted; According to the statistical results, the usage frequency of each cleaning rule is sorted to determine the utilization score; Automatically configure cleaning rules based on utilization scores from high to low.

7. The intelligent data cleaning method based on big data according to claim 6 is characterized in that: The method of sorting the usage frequency of each cleaning rule according to the statistical results and determining the utilization score includes: The utilization score is determined based on the ratio of the number of times the cleaning rule is used to the total number of times all rules are called.

8. An intelligent data cleaning device based on big data, characterized in that: include: An acquisition module is used to acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; A rule determination module is used to determine cleaning rules according to cleaning requirements and cleaning purposes; A cleaning module is used to clean the multi-source business data in the source data table according to the type and category of the multi-source business data using corresponding cleaning rules, and store the cleaned data in the target data table; The rule configuration module is used to count the usage frequency of each cleaning rule and automatically configure the cleaning rules according to the usage frequency of the cleaning rules.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, the steps of the intelligent data cleaning method based on big data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the intelligent data cleaning method based on big data as described in any one of claims 1-7.

Images