Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A mass data cleaning method and device

A data cleaning and mass data technology, applied in the field of data communication, can solve problems such as low operating efficiency, low output efficiency, and lack of unified specifications, and achieve high operating efficiency, consistency, and high efficiency

Inactive Publication Date: 2016-10-12
ADVANCED NEW TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The purpose of this application is to solve the problems in the prior art that the data cleaning code is manually generated, lacks unified specifications, low output efficiency, and low operating efficiency. At the same time, it can accurately locate the source and destination of each piece of dirty data. data quality support

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A mass data cleaning method and device
  • A mass data cleaning method and device
  • A mass data cleaning method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0038] The technical solutions of the present application will be described in further detail below in conjunction with the accompanying drawings and embodiments, and the following embodiments do not limit the present application.

[0039] In this application, a massive data cleaning method such as figure 1 shown, including steps:

[0040] Step 101, configuring data cleaning rule files.

[0041] Specifically, Table 1 provides a specific embodiment of a data cleaning rule file:

[0042]

[0043]

[0044] Table 1

[0045] Taking Table 1 as an example, the data cleaning rule file includes:

[0046] rule_id: rule serial number;

[0047] table_name: the name of the data table, that is, the name of the data table to which the rule belongs;

[0048] bit_offset: rule serial number, which is the binary offset, the function of bit_offset is used to label the data;

[0049] rule_code: Pseudocode of data cleaning rules;

[0050] description: Chinese description of data cleani...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and device for cleaning mass data. The method comprises the steps of first configuring data cleaning rule files, obtaining data cleaning rules corresponding to a to-be-cleaned data table according to a table name of the data cleaning rules, automatically generating cleaning codes to perform cleaning, tagging every to-be-cleaned datum in the cleaning process, analyzing which data cleaning rule the data trigger by tag analysis, and accordingly performing corresponding cleaning processing. The device for cleaning the mass data comprises a data rule configuration module, a data cleaning code generation module, an execution module and an analysis module, and the mass data are cleaned through the mass data cleaning method. The mass data can be effectively cleaned, the efficiency is high, dirty data which are cleaned out are reserved in a classified mode, and sources and whereabouts of every dirty datum can be located precisely.

Description

technical field [0001] The present application belongs to the technical field of data communication, and in particular relates to a method for cleaning massive data. Background technique [0002] With the rapid development of computer technology and communication technology, people can obtain more and more digital information, but at the same time, they need to invest more time in organizing and organizing the digital information. For example, in business systems, some dirty data is often generated due to factors such as code defects, business definition changes, and network delays. For example, the payment time of an order is earlier than the creation time of the order. This is a piece of data that does not conform to business logic. Before performing statistical analysis on the data, these dirty data need to be filtered out to ensure the accuracy of the statistics. Data cleaning is a process of reducing data errors and inconsistencies. The main task is to detect and delet...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/951
Inventor 刘欣
Owner ADVANCED NEW TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com