Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing method and device of real-time computing platform

A real-time computing and data processing technology, applied in the Internet field, can solve problems such as the inability to meet the data processing needs of real-time computing platforms, and achieve the effect of meeting big data processing needs and improving data throughput

Active Publication Date: 2016-12-07
BEIJING QIHOO TECH CO LTD +1
View PDF6 Cites 23 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

For some data sources, the speed at which the receiver reads data is certain. For example, the data to be processed is 200G, the receiver reads 1G data per second, and the real-time computing platform processes the data once a minute, which is equivalent to once for the real-time computing platform. The amount of batch data is 60G, but the memory of the receiver is only 6G, so there is no way to meet the data processing requirements of the real-time computing platform

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device of real-time computing platform
  • Data processing method and device of real-time computing platform
  • Data processing method and device of real-time computing platform

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0069] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0070] figure 1 A flowchart showing a data processing method of a real-time computing platform according to an embodiment of the present invention. Such as figure 1 As shown, the method includes:

[0071] Step S110, receiving a computing task, and reading configuration information of the computing task.

[0072] Step S120, according to the data source information in the configuration information, determine the data source config...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method and device of a real-time computing platform. The method includes the steps that a computing task is received, and configuration information of the computing task is read; according to data source information in the configuration information, a data source configured by the computing task is determined; multiple receivers are dispatched to obtain to-be-processed data from the data source and store the to-be-processed data; according to a processing rule in the configuration information, the to-be-read data is read from the receivers for corresponding processing. The data processing process is conducted according to the configuration information of the computing task received by the real-time computing platform, the data source configured by the computing task is determined, the receivers are dispatched to obtain to-be-processed data from the data source and store the to-be-processed data, and the to-be-processed data is read from the receivers and processed. By means of the scheme, the process is equivalent to the process that multiple independent processes are conducted in parallel to obtain the to-be-processed data from the data source and wait for processing, data throughput of the real-time computing platform is effectively improved, and the big data processing requirement is met.

Description

technical field [0001] The invention relates to the technical field of the Internet, in particular to a data processing method and device for a real-time computing platform. Background technique [0002] In the existing technology, the real-time computing platform can perform data processing based on the computing framework. The statistical processing process of the real-time computing platform is carried out by means of the memory of the computing framework itself. The real-time computing platform uses the receiver to obtain the data to be processed from the data source After that, put the data to be processed directly into the memory, and then perform statistical processing on the data in the memory. For some data sources, the speed at which the receiver reads data is certain. For example, the data to be processed is 200G, the receiver reads 1G data per second, and the real-time computing platform processes the data once a minute, which is equivalent to once for the real-t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/215G06F16/23G06F16/25
Inventor 徐胜国王义辉李铮
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com