Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data extraction method and system based on massive data

A data extraction and mass data technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of high maintenance costs, inflexible settings, rigidity, etc., to reduce the cost and difficulty of operation and maintenance, reduce The effect of extracting the number of tasks and reducing the pressure of extraction

Active Publication Date: 2017-12-05
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF5 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In addition, for massive data, according to the existing technology, a data table with a large amount of data can only be extracted simultaneously by creating multi-task segments
This method will lead to many tasks and high maintenance costs in the later period
[0007] Furthermore, the extraction time stamp field in the existing technology is relatively fixed and rigid (usually the time of creation or modification), and cannot be flexibly set according to business conditions or needs. In the end, we can only expect online research and development to carry out structural transformation of the source table

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data extraction method and system based on massive data
  • Data extraction method and system based on massive data
  • Data extraction method and system based on massive data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] The present invention is described below based on examples, but the present invention is not limited to these examples. In the following detailed description of the invention, some specific details are set forth in detail. The present invention can be fully understood by those skilled in the art without the description of these detailed parts. In order to avoid obscuring the essence of the present invention, well-known methods, procedures, and flow charts are not described in detail. Additionally, the drawings are not necessarily drawn to scale.

[0052]The flow charts and block diagrams in the accompanying drawings illustrate the possible system framework, functions and operations of the systems, methods, and devices of the embodiments of the present invention, and the blocks on the flow charts and block diagrams can represent a module, program segment, or just a segment Code, said modules, program segments and codes are all executable instructions for implementing p...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a data extraction method and system based on mass data. The method comprises the steps of according to set conditions, dividing data in an assigned data source list into dynamic source data and static source data; conducting initialization processing on the static source data to obtain static target source data; configuring an extraction assignment for the dynamic source data, executing the extraction assignment and extracting the dynamic source data to obtain dynamic target data; saving the static target data and the dynamic target data as a corresponding static target data file and a corresponding dynamic target data file respectively, and merging the two files into an assigned data warehouse. The system includes a classification module, a statistic data processing module, a dynamic data processing module and a data storage module. According to the data extraction method and system based on the mass data, the quantity of the extraction assignments is reduced, data extraction efficiency is improved, the cost and difficulty of operation and maintenance in the later period are decreased, and the changes and demands of rapid development of services are met.

Description

technical field [0001] The invention relates to the technical field of database data processing, in particular to a data extraction method and system based on massive data. Background technique [0002] Data Warehouse (DW or DWH for short) is a strategic collection that provides all types of data support for enterprise-level decision-making processes. It is a structured data environment for decision support systems (dss) and online analysis application data sources. The data warehouse system is an information providing platform, which obtains data from the business processing system, and provides users with various means to obtain information and knowledge from the data. Therefore, for some enterprises and institutions, they have built their own data warehouses. [0003] The construction of a data warehouse requires multiple steps, from data extraction, storage to use, every step is crucial. Among them, the data extraction as the first step, the efficiency of data extracti...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/254
Inventor 阎开品葛胜利
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com