Unlock instant, AI-driven research and patent intelligence for your innovation.

Method and system for processing big data

A processing method and processing system technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of low efficiency, large time overhead, high labor cost, etc., to reduce overhead, improve efficiency, and increase the degree of automation Effect

Active Publication Date: 2014-07-23
SHENZHEN INST OF ADVANCED TECH
View PDF4 Cites 11 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The embodiment of the present invention provides a big data processing method and system, aiming to solve the problem of large data processing method provided by the prior art, which not only has huge time consumption, low efficiency, but also high labor cost

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and system for processing big data
  • Method and system for processing big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0051] figure 1 The implementation flow of the big data processing method provided by Embodiment 1 of the present invention is shown, and the details are as follows:

[0052] In step S101, the real-time processing system parses a preset configuration file to obtain data structure information of heterogeneous data sources.

[0053] In this embodiment, the real-time processing system can set the configuration file in advance according to the structural information of the heterogeneous data source to be processed. After receiving the source data stream, the real-time processing system can set the configuration file according to the preset configuration file. The received source data stream is analyzed to obtain the data information of the source data stream.

[0054]Among them, after the user successfully logs in to the main interface of the big data processing system composed of the real-time processing system and the batch processing system, he can manage the data source throu...

Embodiment 2

[0068] figure 2 A specific structural block diagram of the big data processing system provided by Embodiment 2 of the present invention is shown. For the convenience of description, only the parts related to the embodiment of the present invention are shown. The processing system of this big data comprises: real-time data query client 23, real-time processing system 21, batch processing query client 24 and batch processing system 22, described real-time data query client 23 is connected with described real-time processing system 21, so The real-time processing system 21 is connected with the batch processing system 22, and the batch processing system 22 is connected 24 with the batch query client.

[0069] Specifically, the real-time processing system 21 includes: a data structure information acquiring unit 211 , a data structure information sending unit 212 , a real-time processing unit 213 , a first real-time result storage unit 214 and a real-time result sending unit 215 ....

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention belongs to the technical field of information processing, and provides a method and system for processing big data. The method includes the steps that a real-time processing system acquires data structure information of a heterogeneous data source according to a preset configuration file and sends the data structure information of the heterogeneous data source to a batch processing system, and the batch processing system establishes a meta-data table according to the data structure information of the heterogeneous data source; the real-time processing system receives source data flow, conducts real-time processing on the source data flow, writes a real-time processing result in a database of the real-time processing system, and meanwhile sends the real-time processing result to the batch processing system, and the real-time processing result is stored in the meta-data table through the batch processing system; the batch processing system receives batch processing operating instructions, corresponding batch processing tasks are executed according to the batch processing operating instructions, and then an operating result is displayed. According to the method and system, operation for manually preprocessing data is avoided, and preprocessing time cost at the early data processing stage is reduced.

Description

technical field [0001] The invention belongs to the technical field of information processing, and in particular relates to a big data processing method and system. Background technique [0002] Batch processing and real-time processing are two basic methods for data processing in big data analysis. The so-called batch processing is to group jobs (or batches) according to their nature, and then submit them to the computer system in groups (or batches), and the computer will automatically complete the processing and then output the results. Its purpose is to improve system throughput. volume and resource utilization. Today's batch processing systems, such as distributed file systems and parallel databases, have greatly improved the resource utilization and job throughput of the system. They are not only suitable for small data sets stored in memory, but also can be extended to handle huge data set. The so-called real-time processing means that real-time data continuously e...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/24568G06F16/2457
Inventor 范小朋沈慧杨鑫赵东辉须成忠
Owner SHENZHEN INST OF ADVANCED TECH