A method for intelligent cleaning of structured data
A structured data and intelligent cleaning technology, applied in the field of data processing, can solve problems such as file conversion errors, improve data maintainability, facilitate automatic processing of information, and speed up comparison speed
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0045] refer to figure 1 As shown, the embodiment of the present invention provides a structured data intelligent cleaning method, including the following steps:
[0046] S101. Obtain data files to be cleaned based on the local file read-write interface and create a file list;
[0047] S102. Merge all the data files to be cleaned into one file to be cleaned;
[0048] S103, using a hash table to identify the data type and file format contained in the file to be cleaned, and marking the template type to which the identifiable file data belongs;
[0049] S104. Load the file list according to the marked template type, and sequentially perform data cleaning processing of header identification, data verification, format screening, and duplicate checking on the file data;
[0050] S105. Enter the cleaned data into the database one by one using the SQL queryer.
[0051] The steps in the above method are described and illustrated in detail below.
[0052] Understandably, the soluti...
Embodiment 2
[0087] Based on the foregoing embodiment scheme, with reference to figure 2 As shown, Embodiment 2 of the present invention also provides a structured data intelligent cleaning system, the main components of which include a file read and write interface, a multi-file merge module, a screening and verification module, a deduplication module and a database.
[0088]Among them, the user can import a single file or multiple files to be cleaned through the local file read and write interface; if it is a single file, it will directly send the single file to the screening and verification module for cleaning; The files are merged into one file to be cleaned through the multi-file merging module, and then sent to the screening and verification module for cleaning. In this process, the screening and verification module can obtain task parameters for the file data, obtain the main data file in the multi-file, and fill the main data file with other files of the same type as the array to...
Embodiment 3
[0097] Based on the foregoing embodiment scheme, with reference to image 3 As shown, Embodiment 3 of the present invention also provides a specific hardware structure of a structured data intelligent cleaning device. The structured data intelligent cleaning device 3 may include: a memory 32 and a processor 33; each component is coupled together through a communication bus 31 . Understandably, the communication bus 31 is used to realize connection and communication between these components. In addition to the data bus, the communication bus 31 also includes a power bus, a control bus and a status signal bus. But for clarity, in image 3 The various buses are denoted as communication bus 31 in FIG.
[0098] The memory 32 is used to store the structured data intelligent cleaning method program that can be run on the processor 33;
[0099] Processor 33, configured to perform the following steps when running the structured data intelligent cleaning method program:
[0100] St...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More