A large amount of data processing method and system

A large data volume and data processing technology, applied in the field of data processing, can solve problems such as system crash, large data volume data processing, processing delay, etc., and achieve the effect of improving processing capacity, good scalability, and balancing busyness

Inactive Publication Date: 2011-12-21
ALIBABA GRP HLDG LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] The technical problem to be solved by the present invention is to provide a large amount of data processing method and system to solve the problem that the large amount of data cannot be processed within the specified time, resulting in processing delay and finally causing the system to crash

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A large amount of data processing method and system
  • A large amount of data processing method and system
  • A large amount of data processing method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0037] In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0038] The present invention provides a method and system for concurrent or distributed processing of large files. By assigning server concurrency strategies through file naming rules, multiple servers can be deployed to split and process large data files at the same time, which greatly improves It improves the processing capacity of the system and ensures that the system completes the processing of large data volume files within the specified time.

[0039] For example: the sender generates a file FileA (the file has a large capacity, such as 200M) of various categories (such as commodities and orders; there can also be multiple data files of a single category) according to a certain format every 2 minutes, and Send to the recipi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a method and a system for processing large amount of data, so as to solve the problem that the large amount of data cannot be processed within a specified time, causing processing delay and finally causing system collapse. The method includes: assigning a server according to an original file naming rule, and splitting the original file into small files; for each split small file, reassigning a server according to the small file naming rule, and processing the split small files . The invention can deploy multiple servers to split and process files with large amount of data at the same time, which greatly improves the processing capability of the system and ensures that the system completes the processing of files within a specified time. Moreover, the system has very good scalability. When the file size becomes larger or larger, the demand can be met by adding a new server, that is, it can be expanded linearly without purchasing a more advanced server. There is also no need to redeploy previously running servers.

Description

technical field [0001] The invention relates to data processing technology, in particular to a large data volume data processing method and system. Background technique [0002] In many application scenarios, there is often the following data processing process: the sender saves some data in a file in a certain format, then sends the file to the receiver, and the receiver checks the contents of the file after receiving the file Analyze and perform corresponding logical processing. [0003] In the above data processing process, if the file is not very large and the receiver does not have high requirements for processing time, then a single server or single thread can be used for processing at this time. In this case, the system will still function normally, but the receiver may take longer to process the file data. However, if the file is large or there are many files, and the receiver has high requirements for processing time, for example, the receiver requires that the fi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F9/5066G06F9/5055
Inventor 唐益鹏洪文其
Owner ALIBABA GRP HLDG LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap