Multi-cluster cooperative data processing method and system, memory medium and computer device

A data processing system and data processing technology, applied in the field of data processing, can solve problems such as low data exchange efficiency and data redundancy, and achieve the effect of improving data exchange efficiency, reducing the risk of inconsistency, and good scalability

Active Publication Date: 2018-02-09
INDUSTRIAL AND COMMERCIAL BANK OF CHINA
View PDF4 Cites 21 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0007] The embodiment of the present invention provides a multi-cluster collaborative data processing method to overcome the limitations of the cluster constructed by a single technology to deal with business scenario services, as well as the problems of data redundancy and low data exchange efficiency among various heterogeneous syste

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Multi-cluster cooperative data processing method and system, memory medium and computer device
  • Multi-cluster cooperative data processing method and system, memory medium and computer device
  • Multi-cluster cooperative data processing method and system, memory medium and computer device

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0034] In order to make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following describes the embodiments of the present invention in further detail with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and the description thereof are used to explain the present invention, but not as a limitation to the present invention.

[0035] In order to overcome the limitations of clusters constructed by a single technology to respond to business scenario services, as well as the problems of data redundancy and low data exchange efficiency between multiple heterogeneous systems, the present invention proposes a multi-cluster collaborative data processing method. figure 1 It is a schematic flowchart of a multi-cluster collaborative data processing method according to an embodiment of the present invention. Such as figure 1 As shown, the multi-cluster collaborative data processing met...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a multi-cluster cooperative data processing method and system, a memory medium and a computer device. The method comprises the steps of receiving collection data and convertingthe collection data into structured data; establishing a basic data sharing area on a Hadoop cluster and loading the structured data to the basic data sharing area; and cooperatively processing the data in the basic data sharing area through utilization of the Hadoop cluster, a Hadoop online processing cluster and a plurality of distributed database clusters according to a data application scene,thereby providing corresponding data service and finishing a data processing process. According to the method, the system, the memory medium and the computer device, the data redundancy can be reducedthrough establishment of the basic data sharing area.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a multi-cluster cooperative data processing method, system, storage medium and equipment. Background technique [0002] In recent years, with the advent of the era of big data, more and more companies have realized the value of data, and hope to guide their business decisions through data analysis and data mining. Driven by these demands, big data processing technologies applicable to various scenarios have emerged. Among them, the technical ecosystem represented by Hadoop technology and the distributed Database technology is the most representative. The share nothing architecture is adopted, each node uses independent local resources, and data is generally distributed according to the distribution key Hash, and data exchange between nodes is realized through the network to achieve linear expansion of the system and efficiently process PB-level data. [0003] Al...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F16/24G06F16/258G06F16/27
Inventor 杨晨旭王备蔡良良阳维迅
Owner INDUSTRIAL AND COMMERCIAL BANK OF CHINA
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap