Oracle data batch collection method based on ROWID

A collection method and data collection node technology, applied in the Oracle database field, can solve problems such as poor Oracle data, inability to collect data quality, and null values ​​in all columns, and achieve simple technology, high data collection performance, and low development costs.

Inactive Publication Date: 2018-10-16
INSPUR QILU SOFTWARE IND
View PDF4 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The JDBC method is simple and easy to use, but these products have certain requirements on data quality, and some Oracle data with poor data quality cannot be collected, such as the existence of a large amount of duplicate data, the lack of primary keys and indexes, and the presence of null values ​​​​in all columns, etc.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Oracle data batch collection method based on ROWID

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0024] The operation steps of the ROWID-based Oracle data batch collection method are as follows:

[0025] Step 1) Generate SQL statements by querying the ROWID starting from each SQL through the JDBC interface; each of the generated SQL statements collects n ten thousand pieces of data, then sorts the ROWIDs, and obtains the first and n+1 , 2n+1... ROWIDs; the SQL statement for obtaining the ROWID is executed in parallel mode, and the SQL statement is executed through JDBC, and the result set is an ordered ROWID queue. The ROWID queue is used to generate batches of SQL statements.

[0026] The SQL statement is as follows: tName is the table name of data collection; pSize is the batch size of data collection; pCount is the number of parallel threads for executing this SQL.

[0027] SELECT RID AS "ROWID" FROM (

[0028] SELECT a.*, ROWNUM rnum FROM (

[0029] SELECT / *+ Parallel(pCount) * / ROWID AS RID FROM tName ORDER BY ROWID)

[0030] a) WHERE MOD(rnum, pSize) = 1;

[0...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses an Oracle data batch collection method based on ROWID. According to the method, through interaction between an Oracle database and a JDBC interface, a batch of SQL statements are generated by the ROWID, each SQL statement is used for collecting a part of data in a database table, and a data set collected by all the SQL statements is the complete data. Compared with the prior art, the Oracle data batch collection method based on the ROWID has the advantages that on the basis of the JDBC interface, there is no intrusion to the Oracle database, the technology is simple, the method is easy to implement, low in development cost, flexible and diverse, expandability is achieved, there are no requirements for data quality, distributed batch operations are adopted, the dataacquisition performance is extremely high, and the method has wide applicable scenes.

Description

technical field [0001] The invention relates to the technical field of Oracle databases, in particular to a ROWID-based Oracle data batch collection method. Background technique [0002] Oracle data acquisition is a requirement often encountered by software designers and developers, such as importing Oracle data from old systems into new systems, summarizing Oracle data from subsystems into data center systems, importing external Oracle data into big data platforms, etc. . [0003] At present, there are many open source Oracle collection tools, such as Sqoop, Kattle, DataX, etc., all of which use JDBC to collect data. The JDBC method is simple and easy to use, but these products have certain requirements on data quality, and some Oracle data with poor data quality cannot be collected, such as the existence of a large amount of duplicate data, the lack of primary keys and indexes, and the presence of null values ​​​​in all columns. Contents of the invention [0004] The t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 刘福才胡清肖雪王建华
Owner INSPUR QILU SOFTWARE IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap