An intelligent cleaning data method, device and electronic equipment based on big data

Abstract

The present application relates to a kind of based on big data intelligent cleaning data method, device and electronic equipment, belong to big data technical field, wherein, the method includes: obtaining multi-source business data, according to multi-source business data to establish source data table, and according to source data table to establish target data table with mapping relationship;According to cleaning demand and cleaning purpose, determine cleaning rule;According to the type and category of multi-source business data, using corresponding cleaning rule to clean multi-source business data in source data table, and the data after cleaning is stored in target data table;The use frequency of each cleaning rule is counted, and automatically configure cleaning rule according to cleaning rule use frequency.The present application solves the technical problem that data is not automatically cleaned in the prior art, and a large amount of manpower and time cost needs to be invested, thereby leading to the technical problem of low data cleaning efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to an intelligent data cleaning method and device based on big data and an electronic device. BACKGROUND

[0002] With the continuous development of informatization, the application of cloud computing and big data has been highly valued by governments, scientific research institutions and industry manufacturers. In order to improve the comprehensive utilization rate of data resources and solve the problems of "information islands", non-unified, non-standard and incomplete data in traditional business systems, big data centers need to be established in various industries.

[0003] A big data center is a powerful tool for business experts to analyze various types of massive business data. However, the original data standards are different and cannot be directly used, and the data needs to be cleaned to improve the data usability before being used. The existing technology does not automatically clean the data, and a large number of personnel are needed to manually clean the data. This will increase the user's manpower and time cost, and is time-consuming and laborious with low efficiency. SUMMARY

[0004] Therefore, it is necessary to provide an intelligent data cleaning method and device based on big data and an electronic device to solve the technical problem of low data cleaning efficiency caused by the lack of automatic data cleaning in the prior art and the need for a large amount of manpower and time cost.

[0005] To solve the above problems, the present application provides an intelligent data cleaning method based on big data, comprising:

[0006] Obtaining multi-source business data, establishing a source data table according to the multi-source business data, and establishing a target data table with a mapping relationship according to the source data table;

[0007] According to the cleaning requirements and the cleaning purposes, the cleaning rules are determined;

[0008] According to the type and category of the multi-source business data, the corresponding cleaning rules are used to clean the multi-source business data in the source data table, and the cleaned data is stored in the target data table;

[0009] The use frequency of each cleaning rule is counted, and the cleaning rules are automatically configured according to the cleaning rule use frequency.

[0010] In one possible implementation, the source data table is established according to the multi-source business data, and the target data table with a mapping relationship is established according to the source data table, comprising:

[0011] According to the multi-source business data, the structure of the source data table is determined, and the type and category of the multi-source business data corresponding to each field in the source data table are defined.

[0012] According to the table mapping, the field mapping and the data conversion rule, a mapping relationship between the source data table and the target data table is established.

[0013] In a possible implementation, before determining the cleaning rule, the method further includes:

[0014] The multi-source business data is classified and labeled to determine the category and type of the multi-source business data.

[0015] The labeled multi-source business data is preprocessed to remove data spaces at the beginning and end, convert the letter case and convert the full-width Chinese character to half-width Chinese character.

[0016] In a possible implementation, the cleaning rule at least includes:

[0017] According to the data classification and labeling, a cleaning rule for deleting or separately storing the repeated data is established.

[0018] The date and time format data is pre-set and standardized converted.

[0019] The numerical format data is pre-set and standardized converted.

[0020] The different types of data are pre-set and standardized converted according to the type dictionary table according to the requirement.

[0021] The special symbols in the string are replaced and decomposed.

[0022] The data to be extracted in the string is extracted according to the regular expression.

[0023] In a possible implementation, the multi-source business data in the source data table is cleaned by using the corresponding cleaning rule, and the cleaned data is stored in the target data table, including:

[0024] An abnormal data storage table is established.

[0025] The multi-source business data in the source data table is cleaned by using the corresponding cleaning rule, and the multi-source business data successfully cleaned is stored in the target data table.

[0026] And the data failed to be cleaned is stored in the abnormal data storage table.

[0027] In a possible implementation, the use frequency of each cleaning rule is counted, and the cleaning rule is automatically configured according to the use frequency of the cleaning rule, including:

[0028] Based on each multi-source business data cleaning process, the configured number of the cleaning rule involved is counted.

[0029] According to the statistical result, the usage frequency of each cleaning rule is sorted, and a utilization score is determined;

[0030] The cleaning rules are automatically configured from high to low according to the utilization scores.

[0031] In a possible implementation, the usage frequency of each cleaning rule is sorted according to the statistical result, and the utilization score is determined, including:

[0032] The utilization score is determined according to a ratio of the usage times of the cleaning rule to total calling times of all rules.

[0033] In a second aspect, the present application further provides an intelligent cleaning data device based on big data, comprising:

[0034] An acquisition module is configured to acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table having a mapping relationship according to the source data table;

[0035] A rule determination module is configured to determine cleaning rules according to cleaning requirements and cleaning purposes;

[0036] A cleaning module is configured to clean the multi-source business data in the source data table according to the types and categories of the multi-source business data by using corresponding cleaning rules, and store the cleaned data into the target data table;

[0037] A rule configuration module is configured to statistically determine the usage frequency of each cleaning rule, and automatically configure the cleaning rules according to the usage frequency of the cleaning rules.

[0038] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory;

[0039] The memory has stored a computer readable program which can be executed by the processor;

[0040] The processor executes the computer readable program to realize the steps in the intelligent cleaning data method based on big data.

[0041] In a fourth aspect, the present application further provides a computer readable storage medium, which stores one or more programs which can be executed by one or more processors to realize the steps in the intelligent cleaning data method based on big data.

[0042] The beneficial effects of the present application are: firstly, acquiring multi-source business data, establishing a source data table according to the multi-source business data, and establishing a target data table with a mapping relationship according to the source data table; and determining the cleaning rules according to the cleaning requirements and the cleaning purposes; according to the type and category of the multi-source business data, the corresponding cleaning rules are used to clean the multi-source business data in the source data table, and the cleaned data is stored in the target data table; the purpose of automatically cleaning the data by the cleaning rules is realized. Finally, the use frequency of each cleaning rule is counted, and the cleaning rules are automatically configured according to the cleaning rule use frequency, the use frequency is determined through the iteration and accumulation of the cleaning standards of different types of data, so that the intelligent rule library automatically matches the data cleaning rules according to the data classification and the use frequency, without manually configuring the data cleaning rules by the user, the user can edit the automatically configured cleaning rules, which maximally reduces the manual data cleaning, and achieves the purpose of automatic data cleaning. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A method flowchart of an embodiment of the intelligent data cleaning method based on big data provided by the present application is provided.

[0044] Figure 2 A schematic diagram of an implementation mode of the intelligent data cleaning method based on big data provided by the present application is provided.

[0045] Figure 3 In the intelligent data cleaning method based on big data provided by the present application, a method flowchart of an embodiment of step S101 is provided.

[0046] Figure 4 In the intelligent data cleaning method based on big data provided by the present application, a method flowchart of an embodiment of step S103 is provided.

[0047] Figure 5 In the intelligent data cleaning method based on big data provided by the present application, a method flowchart of an embodiment of step S104 is provided.

[0048] Figure 6 A schematic diagram of an embodiment of the intelligent data cleaning device based on big data provided by the present application is provided.

[0049] Figure 7 A running environment schematic diagram of an embodiment of the electronic device provided by the present application is provided. DETAILED DESCRIPTION

[0050] The preferred embodiments of the present application will be specifically described below in combination with the drawings, wherein the drawings form a part of the present application, and are used to illustrate the principles of the embodiments of the present application, and are not used to limit the scope of the present application.

[0051] One specific embodiment of the present application discloses a big data-based intelligent cleaning data method, please refer to Figure 1 and Figure 2 , comprising:

[0052] S101, acquiring multi-source business data, establishing a source data table according to the multi-source business data, and establishing a target data table with a mapping relationship according to the source data table;

[0053] S102, determining cleaning rules according to cleaning requirements and cleaning purposes;

[0054] S103, cleaning the multi-source business data in the source data table according to the type and category of the multi-source business data by using the corresponding cleaning rules, and storing the cleaned data into the target data table;

[0055] S104, counting the usage frequency of each cleaning rule, and automatically configuring the cleaning rules according to the usage frequency of the cleaning rules.

[0056] In the embodiment, first, multi-source business data is acquired, a source data table is established according to the multi-source business data, and a target data table with a mapping relationship is established according to the source data table; cleaning rules are determined according to cleaning requirements and cleaning purposes; the multi-source business data in the source data table is cleaned according to the type and category of the multi-source business data by using the corresponding cleaning rules, and the cleaned data is stored into the target data table; the purpose of automatically cleaning data by cleaning rules is achieved. Finally, the usage frequency of each cleaning rule is counted, and the cleaning rules are automatically configured according to the usage frequency of the cleaning rules. Through the iteration and accumulation of cleaning standards for different types of data, the usage frequency is determined, so that according to the data classification of the user-defined data table and data field, the intelligent rule library automatically matches the data cleaning rules according to the data classification and usage frequency, without the need for the user to manually configure the data cleaning rules. The user can edit the automatically configured cleaning rules, which maximally reduces manual data cleaning and achieves the purpose of automatic data cleaning.

[0057] In step S101, different data collection methods are used to acquire multi-source business data according to different sources of business data, and the data is persisted to a local database. The scope and type of the business are not limited.

[0058] In step S102, the cleaning rule library refers to a system or module that centrally stores and manages data cleaning rules. It contains a series of predefined, reusable rules designed to clean and transform raw data to improve data quality and consistency. Before determining the cleaning rules, a comprehensive quality assessment of the source data is first conducted to identify and record problems existing in the data. Key issues include null values, format errors, duplicate data, outliers, etc. Statistical methods are used to understand the overall situation of the data through data summary statistics and distribution analysis.

[0059] In some embodiments, the source data table is established according to the multi-source business data, and the target data table with a mapping relationship is established according to the source data table, please refer to Figure 3 , including:

[0060] S301, according to the multi-source business data, determine the structure of the source data table, and define the type and kind of multi-source business data corresponding to each field in the source data table;

[0061] S302, according to the table mapping, field mapping and data conversion rule, establish the mapping relationship between the source data table and the target data table.

[0062] In this embodiment, the source data table is used to store the original data obtained from multiple business systems or data sources. Each source data table corresponds to a specific business data source, usually contains multiple fields, and each field represents a property of the data. According to the characteristics and needs of the data source, the data table structure is designed, including table name, field name, field type, etc.

[0063] The target data table is a target table that stores processed and cleaned data, usually used for analysis, reporting and decision support. The data cleaned and converted is loaded from the source data table to the target data table.

[0064] Further, the mapping relationship between the source data table and the target data table includes but is not limited to:

[0065] 1. Table mapping design

[0066] Analyze business requirements and understand business needs, such as which reports or analysis results need to be output.

[0067] Data source classification, determine the source business system, and classify according to business logic, such as sales, customers, inventory, etc.

[0068] Summarize the data structure and record the structure of the source data table in detail, including table name, field name, field type and sample data.

[0069] 2. Field mapping design

[0070] One-to-one / multiple-to-one, determine how each field in the source data table maps to the fields of the target data table. Sometimes one source field can correspond to multiple target fields, and vice versa.

[0071] 3. Data conversion rules

[0072] According to the field mapping design, list the required conversion rules. For example, it may be necessary to convert the date format from "DD / MM / YYYY" to "YYYY-MM-DD".

[0073] Determine the rules for data standardization, such as removing invalid characters, adjusting case, unifying units, etc.

[0074] In one specific embodiment, the relationship between data tables, data fields and field cleaning rules is manually defined in the intelligent rule library, and the field data classification is identified. The mapping relationship between the source data table and the target data table is established, the mapping relationship between the source data table field and the target data table field is established, the cleaning rules of the data table field are configured, and the field data classification is identified. For example: the source data table AA and the target data table BB establish a mapping relationship, the A1, A2, A3 fields in the AA table and the B1, B2, B3 fields in the BB table establish a mapping relationship, the data of the A1 field is written into the B1 field, and the others are similar. Configure the cleaning rules of B1 field, you can configure automatic data completion and other cleaning rules for B1 field. Identify the data type of B1 field, such as date type data, etc. When the field is not configured with data cleaning rules, the system will match the default cleaning rules according to the field classification.

[0075] In some embodiments, before determining the cleaning rules, it further includes:

[0076] Classify and label the multi-source business data to determine the type and category of the multi-source business data.

[0077] Preprocess the labeled multi-source business data to remove leading and trailing spaces, letter case conversion, and full-width Chinese character conversion to half-width.

[0078] In this embodiment, in the data preprocessing process, through the business data preprocessing adaptive configuration function, according to the data mapping relationship setting of the data cleaning rule library, the business data of different parameter characteristics is classified and labeled, and the leading and trailing spaces of the data, the letter case conversion, and the full-width Chinese character conversion to half-width are removed. The data that fails in preprocessing is stored separately.

[0079] In a specific embodiment, the data identifier (table name) is matched against the data identifier record in the cleaning rule base. If no match is found, the preprocessing is deemed to have failed. If the match is successful, the field is matched according to the field mapping relationship of the table in the cleaning rule base. If the match fails (the success criteria are given by the cleaning rule base), the preprocessing is deemed to have failed. If the match is successful, general data processing is performed, such as removing leading and trailing spaces, and the preprocessing is completed.

[0080] Furthermore, the cleaning rules include at least:

[0081] Based on data classification labels, establish cleaning rules for deleting or storing duplicate data separately;

[0082] Perform preset standardization conversion on date and time format data;

[0083] Perform preset standardization conversion on digital format data;

[0084] For different types of data, perform pre-defined standardized transformations based on the type dictionary table according to requirements;

[0085] Replace and decompose special characters in a string;

[0086] Extract the data that needs to be extracted from the string using regular expressions.

[0087] In some embodiments, the process involves using corresponding cleaning rules to clean the multi-source business data in the source data table and storing the cleaned data in the target data table. Please refer to [link to relevant documentation]. Figure 4 ,include:

[0088] S401. Establish an abnormal data storage table;

[0089] S402. Use the corresponding cleaning rules to clean the multi-source business data in the source data table, and store the successfully cleaned multi-source business data in the target data table.

[0090] S403, and store the data that failed to be cleaned into the abnormal data storage table.

[0091] In this embodiment, successfully cleaned data is stored in a specially designed abnormal data storage table. This not only improves the accuracy of the cleansing process, but also further enhances the effectiveness and accuracy of the data cleansing by allowing subsequent manual processing of the data in the abnormal data storage table.

[0092] In some embodiments, the frequency of use of each cleaning rule is statistically analyzed, and cleaning rules are automatically configured based on the frequency of use. Please refer to [link to relevant documentation]. Figure 5 ,include:

[0093] S501, based on each time of multi-source business data cleaning process, the configured number of cleaning rules involved is counted;

[0094] S502, according to the statistical result, the frequency of use of each cleaning rule is sorted, and the utilization score is determined;

[0095] S503, according to the utilization score from high to low, the cleaning rule is automatically configured.

[0096] In this embodiment, the number of times of configuring the cleaning rule in the intelligent rule library by the user is counted according to the cleaning rule category, the data cleaning rule is sorted according to the configuration number, and the utilization score of the cleaning rule is given according to the ranking. The cleaning rule ranked first is scored as 100 points, the second is scored as 99 points, and all data cleaning rules are scored in this way. The utilization score provides a basis for judging the data intelligent cleaning.

[0097] According to the data classification of the user-defined data table and data field, the intelligent rule library automatically matches the data cleaning rule according to the data classification and utilization score standard, without manual configuration of the data cleaning rule by the user. The user can edit the automatically configured cleaning rule.

[0098] It should be noted that the intelligent rule library is manually configured, and the cleaning rule or data mapping relationship can be added, or the existing cleaning rule and data mapping relationship can be viewed, edited and deleted.

[0099] Based on the above-mentioned intelligent cleaning data method based on big data, the embodiment of the application further provides an intelligent cleaning data device 600 based on big data, comprising:

[0100] The acquisition module 610 is configured to acquire multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table;

[0101] The rule determination module 620 is configured to determine the cleaning rule according to the cleaning requirement and the cleaning purpose;

[0102] The cleaning module 630 is configured to clean the multi-source business data in the source data table according to the type and category of the multi-source business data, and store the cleaned data into the target data table;

[0103] The rule configuration module 640 is configured to count the frequency of use of each cleaning rule, and automatically configure the cleaning rule according to the frequency of use of the cleaning rule.

[0104] As Figure 7As shown, based on the above intelligent cleaning data method based on big data, the application also correspondingly provides an electronic device, which can be a mobile terminal, a desktop computer, a notebook, a palm computer, a server and other computing electronic devices. The electronic device includes a processor 710, a memory 720 and a display 730. Figure 7 Only part of the components of the electronic device are shown, but it should be understood that all the shown components are not required, and more or less components can be alternatively implemented.

[0105] The memory 720 can be an internal storage unit of the electronic device in some embodiments, such as a hard disk or a memory of the electronic device. The memory 720 can also be an external storage device of the electronic device in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device. Further, the memory 720 can include both the internal storage unit and the external storage device of the electronic device. The memory 720 is used to store application software and various data installed on the electronic device, such as program codes installed on the electronic device. The memory 720 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 720 stores an intelligent cleaning data program based on big data 740, which can be executed by the processor 710 to implement the intelligent cleaning data method based on big data of the embodiments of the application.

[0106] The processor 710 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, used to run program codes or process data stored in the memory 720, such as to execute the intelligent cleaning data method based on big data, etc.

[0107] The display 730 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 730 is used to display information of the electronic device based on big data and to display a visual user interface. The components 710-730 of the electronic device communicate with each other through a system bus.

[0108] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiment methods can be completed by instructing the relevant hardware by a computer program, and the program can be stored in a computer readable storage medium. The computer readable storage medium is a disk, an optical disk, a read-only memory, a random access memory, etc.

[0109] The above description is merely preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A big data-based intelligent cleaning data method, characterized in that, The method comprises the following steps: obtaining multi-source business data, establishing a source data table according to the multi-source business data, and establishing a target data table with a mapping relationship according to the source data table; determining cleaning rules according to cleaning requirements and cleaning purposes; cleaning the multi-source business data in the source data table according to the types and categories of the multi-source business data by using corresponding cleaning rules, and storing the cleaned data in the target data table, including: establishing an abnormal data storage table; cleaning the multi-source business data in the source data table by using corresponding cleaning rules, storing the multi-source business data that is successfully cleaned in the target data table; and storing the data that fails to be cleaned in the abnormal data storage table; statistically analyzing the frequency of use of each cleaning rule, and automatically configuring the cleaning rules according to the frequency of use of the cleaning rules, including: statistically analyzing the number of times that each cleaning rule is configured based on each multi-source business data cleaning process; sorting the frequency of use of each cleaning rule according to the statistical results, and determining a utilization score; and automatically configuring the cleaning rules from high to low according to the utilization score. 2.The big data-based intelligent cleaning data method according to claim 1, wherein, The method comprises the following steps: determining the structure of the source data table according to the multi-source business data, and defining the types and categories of the multi-source business data corresponding to each field in the source data table; establishing the mapping relationship between the source data table and the target data table according to table mapping, field mapping, and data conversion rules. 3.The big data based intelligent cleaning data method according to claim 1, wherein, Before determining the cleaning rules, the method further comprises the following steps: classifying and marking the multi-source business data, and determining the categories and types of the multi-source business data; preprocessing the marked multi-source business data, and removing data spaces at the beginning and end, converting letters between upper and lower cases, and converting full-width Chinese characters to half-width Chinese characters.

4. The big data based intelligent cleaning data method according to claim 3, wherein, The cleaning rules at least include the following: establishing a cleaning rule for deleting or separately storing repeated data according to data classification and marking; performing preset standardization conversion on date and time format data; performing preset standardization conversion on numerical format data; performing preset standardization conversion on different types of data according to a type dictionary table according to requirements; replacing and decomposing special symbols in a string; extracting data in a string according to a regular expression. 5.The big data based intelligent cleaning data method according to claim 1, wherein, The method comprises the following steps: determining the utilization score according to the ratio of the number of times that each cleaning rule is used to the total number of times that all rules are called.

6. A big data-based intelligent cleaning data device, characterized in that, The method comprises the following steps: an acquisition module, configured to obtain multi-source business data, establish a source data table according to the multi-source business data, and establish a target data table with a mapping relationship according to the source data table; a rule determination module, configured to determine cleaning rules according to cleaning requirements and cleaning purposes; a cleaning module, configured to clean the multi-source business data in the source data table according to the types and categories of the multi-source business data by using corresponding cleaning rules, and store the cleaned data in the target data table, including: establishing an abnormal data storage table; cleaning the multi-source business data in the source data table by using corresponding cleaning rules, storing the multi-source business data that is successfully cleaned in the target data table; and storing the data that fails to be cleaned in the abnormal data storage table; The rule configuration module is configured to count the usage frequency of each cleaning rule and automatically configure the cleaning rule according to the usage frequency of the cleaning rule, comprising: counting the configured times of the cleaning rule involved in each multi-source business data cleaning process; sorting the usage frequency of each cleaning rule according to the counting result to determine the utilization score; and automatically configuring the cleaning rule from high to low according to the utilization score.

7. An electronic device, comprising: Comprise: a processor and a memory; the memory stores a computer readable program that can be executed by the processor; the processor executes the computer readable program to realize the steps in the intelligent big data based data cleaning method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores one or more programs that can be executed by one or more processors to realize the steps in the intelligent big data based data cleaning method according to any one of claims 1-5.

Images