Method and device for sequential parallel data processing

A technology for data processing and scheduled processing, applied in the field of data processing, can solve the problems of inability to process records, slow running speed, and slow down processing speed, so as to avoid single point of failure, speed up processing, and improve scalability.

Active Publication Date: 2021-03-16
ADVANCED NEW TECH CO LTD
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] It can be seen that if the running speed of the unified sorting function module is slow, the overall processing speed of this step will be dragged down, and the processing speed can only be improved by upgrading the running environment of the unified sorting function module; , all records will not be processed
That is, the process has a single-node bottleneck that does not scale in parallel

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method and device for sequential parallel data processing
  • Method and device for sequential parallel data processing
  • Method and device for sequential parallel data processing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0021] The embodiment of this application proposes a new method for parallel data processing in sequence, records the out-of-order data to be processed in at least two groups for the first parallel sorting, and the results of the first parallel sorting for each group Distributed to the queue corresponding to each processing function module, and then the records in the queue corresponding to each processing function module are sorted in parallel for the second time, so that each processing function module can receive the records arranged in order Records, so that the sorting of records is completed in parallel, avoiding a single point of failure, and providing parallel scalability by adding groups, so as to solve the problems existing in the prior art.

[0022] In the embodiment of the present application, a group can be regarded as a set composed of some records, and the union of all groups includes all records. The out-of-order records in each group can be stored in a storage...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method for performing parallel data processing in order. Parallel data processing is completed through N processing function modules, wherein N is a natural number greater than 1. The method comprises the steps that disordered records are received in M groups, and the records in each group are ordered, wherein M is a natural number greater than 1; the ordered records in each group are distributed to N queues corresponding to the group; each group corresponds to identical N queues; the N queues of each group correspond to the N processing function modules, and each processing function module corresponds to M queues; and the records in the M queues corresponding to each processing function module are output to the corresponding processing function module in order. Through the technical scheme, parallel ordering of the disordered records is realized, it is easy to increase processing speed by adding groups when ordering speed is low, the extensibility in the dataprocessing process is improved, and a single point of failure is avoided.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a method and device for performing parallel data processing sequentially. Background technique [0002] With the in-depth development of information technology, almost all behaviors of people are constantly generating data. The collection, analysis and application of data has become the key for enterprises to improve their competitiveness in the current business environment. The rapid increase in the amount of data and the diversification of analysis methods make the data processing process more and more complicated. [0003] The data processing process can be carried out in multiple steps, and each step completes a part of data processing. In order to cope with the huge amount of computation and the requirements for processing speed, each step is usually operated in parallel by multiple processing function modules to process the input records (units of da...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F9/50
CPCG06F9/5038
Inventor 赵星
Owner ADVANCED NEW TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap