Processing system and processing method of large-scale concurrent data stream

A technology for processing systems and data streams. It is applied in the field of data processing and can solve problems such as improving processing efficiency and failing to meet the data characteristics requirements of stream processors.

Inactive Publication Date: 2011-09-28
UNIV OF SHANGHAI FOR SCI & TECH
View PDF5 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, since the same stream processor core can only run the same core program at the same time, and the thousands of data flow units of the large-scale concurrent data flow coming at the same time do not necessarily have the same processing characteristics, it cannot satisfy the requirements of the flow processing system. The data characteristics required by the processor for high-performance processing cannot enable the stream processor to exert its maximum capabilities, thereby improving processing efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Processing system and processing method of large-scale concurrent data stream
  • Processing system and processing method of large-scale concurrent data stream
  • Processing system and processing method of large-scale concurrent data stream

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The embodiments of the present invention will be described in further detail below in conjunction with the accompanying drawings, but the present embodiments are not intended to limit the present invention, and any similar structures and similar changes of the present invention should be included in the protection scope of the present invention.

[0055] In order to further describe in detail a large-scale concurrent data stream processing system according to an embodiment of the present invention, this specification makes the following settings:

[0056] Setting 1 (data stream unit), a data stream unit is a nine-tuple DSU (id, sno, segno, seq, t, type, prog, data, odata), where DSU is the data stream unit, and id is the DSU and the id is unique, sno is the data flow number of the DSU, segno is the data flow segment number of the DSU, and seq is the unit sequence number of the DSU in the segno data flow segment, which is used to indicate that it is in the The position i...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a processing system and a processing method of a large-scale concurrent data stream, relates to the technical field of data and aims to solve the technical problem of low processing efficiency of a stream processor. The system comprises a data stream unit (DSU) buffer area, a DSU cluster queue pool, a DSU mapping table, a stream processor pool, a data stream reading part, a DSU cluster allocation part, a task scheduling part and a computation post-processing part, wherein the stream processor tool consists of a plurality of graphics processing units (GPU); the data stream reading part is used for writing the concurrent data stream into the DSU buffer area; the DSU cluster allocation part is used for classifying data stream units which are currently processed in the DSU buffer area; the task scheduling part is used for loading a ready queue in the DSU cluster queue pool to the GPUs in the stream processor pool and carrying out stream computation; and the computation post-processing part is used for returning a computation result of the GPUs to the data stream. The system provided by the invention can improve the processing efficiency of the stream processor.

Description

technical field [0001] The invention relates to data processing technology, in particular to a large-scale concurrent data stream processing system and a processing method thereof. Background technique [0002] With the rapid development of information technology and the popularization of Internet technology applications, the network has become an important part of people's daily life. In recent years, the gradual development of 3G network and the application of Internet of Things technology has brought great convenience to people's life and work. However, one of the core application keys of these new technologies is the problem of large-scale concurrent data stream processing. The so-called data flow is a continuous information flow composed of data units with certain characteristics sent from one node to another node. The processing and analysis of data flow requires that the processing node has certain real-time processing capabilities. People can solve real-time proces...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/38G06F9/48
Inventor 陈庆奎那丽春刘伯承王海峰郝聚涛霍欢赵海燕庄松林丁晓东
Owner UNIV OF SHANGHAI FOR SCI & TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap