Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A method and system for batch processing of data

A technology of batch processing and data, applied in the field of data processing, can solve the problems that cannot be loaded and processed in parallel by multiple servers, and the efficiency and performance are difficult to achieve satisfaction.

Inactive Publication Date: 2018-12-28
CHINA CONSTRUCTION BANK
View PDF5 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In view of this, the present invention provides a method and system for batch processing of data, which is used to solve the problems in the prior art due to the single-database single-table mechanism and resource limitations, which cannot be loaded and processed in parallel by multiple servers, but can only be processed serially. It is difficult to achieve satisfactory results in terms of efficiency and performance to complete the loading and processing of all data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and system for batch processing of data
  • A method and system for batch processing of data
  • A method and system for batch processing of data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0055] The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a batch processing method of data, which is characterized in that: when receiving a processing request for a target data set, each target data contained in the target data setis analyzed to determine a cluster to which each target data belongs; when the processing request for the target data set is received, each cluster to which the target data belongs is determined; whenthe processing request for the target data set is received, each cluster to which the target data belongs is determined; each target data contained in each cluster is segmented in parallel to obtaineach slice data set; a table and a database to which each slice data in the respective slice data sets belongs are determined; each table in each database is processed in parallel with each slice data. According to the above method, target data in the target dataset is sliced into different tables in different databases, as the parallel process is carried out through the load of multiple servers,the problem that in the prior art, due to the mechanism of a single library and a single table and the resource limitation, the load and processing of all data can not be carried out through the parallel loading and proces of multiple servers, and the loading and processing flow of all data can only be completed through a serial mode, and the processing efficiency is low is avoided.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a batch processing method and system for data. Background technique [0002] The bank's IT (Information Technology) system usually involves the storage and processing of massive data. The daily full or incremental data files need to be loaded and processed regularly. In the existing technology, the centralized loading and processing is done through a single database and a single table. Regardless of the size of the data file, by adjusting the storage space of the server, data is always appended or updated to the same table in batches. [0003] The inventor conducted research on the existing data batch processing method and found that due to the single-database single-table mechanism and resource constraints, it cannot be loaded and processed in parallel by multiple servers, and all data can only be loaded and processed in a serial manner It is difficult to achieve satisf...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
Inventor 陈仲宇尹涛刘新阳朱建兵
Owner CHINA CONSTRUCTION BANK
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com