Data extraction method and system

A data extraction and data technology, applied in the field of data processing, can solve the problems that the data processing logic cannot be configured, the access pressure of the database server is not very ideal, etc.

Active Publication Date: 2017-11-21
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF11 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The specific workflow is as follows: business system data source preparation, including sql server, mysql, oracle and other relational databases; use sqoop120 to extract business data, but all query conditions are placed in the database for execution, which has caused a lot of problems for database server access Pressure; write extracted data to hdfs130
[0005] In the above solution, the data processing logic cannot be configured; although concurrent data extraction is supported, it is not ideal for reducing the access pressure of the database server with a large amount of data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data extraction method and system
  • Data extraction method and system
  • Data extraction method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted.

[0032] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided in order to give a thorough understanding of embodiments of the present disclosure. However, those skilled in ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a data extraction method and system, and belongs to the technical field of data processing. The method comprises the steps of extracting data in a data source needed to be collected currently in a preset mode from a database server; collecting preset query conditions from the data source; configuring part of or all of the preset query conditions into an XML file written according to a preset rule; automatically analyzing the XML file, and reading the configured preset query conditions; and filtering the data in the data source according to the preset query conditions. According to the method and the system, the configurability of data logic processing in a big data processing process can be realized.

Description

technical field [0001] The present disclosure relates to data processing technology, in particular to a data extraction method and system. Background technique [0002] In the EBS integrated middleware system, the data generated by the business system needs to be imported into the EBS intermediate table in a timely, accurate and complete manner according to certain rules, and the first step is to extract the required data from various data sources. However, if too many query conditions are added during the extraction process, it will cause a lot of access pressure on the database server. [0003] figure 1 The implementation scheme of existing data extraction is described, including business table 110, sqoop120 and hdfs (Hadoop Distributed File System, distributed file system) 130. Among them, sqoop is mainly used to transfer data between hadoop (live) and traditional databases (mysql, postgresql, etc.). [0004] The specific workflow is as follows: business system data so...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2458G06F16/254
Inventor 王军涛张丽
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap