A large-scale data processing method and system
A large-scale data and processing system technology, applied in the field of computer networks, can solve the problems of data transmission speed and quality affecting user experience, performance that cannot meet the needs of large-scale data processing, and data reading and writing pressure, etc., to solve large-scale Effects of data storage, disk utilization improvement, and server cost saving
Active Publication Date: 2016-08-17
BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF3 Cites 0 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
User data is widely distributed in many places. For users, data storage and backup that are not well managed pose hidden dangers to business operations. The speed and quality of data transmission affect user experience. In addition, with the development of cloud services With the gradual rise and promotion, the processing needs of large-scale data storage, statistics or analysis have become urgent problems to be solved
However, the existing data processing systems and methods cannot meet the processing requirements of large-scale data due to the impact of performance. For example, if the existing data processing systems and methods are directly applied to the storage of large-scale data, it will bring unbearable Data read and write pressure
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0042] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.
[0043] At first the processing system of large-scale data provided by the present invention is described, as figure 1 As shown, the system may include: a flow collection subsystem 100 and a flow processing subsystem 200 .
[0044] The traffic collection subsystem 100 is configured to collect data traffic, and mirror the collected data traffic to the server cluster in the traffic processing subsystem 200 .
[0045] Specifically, it may include: a traffic collection unit 110 for collecting data traffic and mirroring the collected data traffic, and may further include: a distribution processing unit 120 for splitting mirrored traffic into sub-flows by using load balancing technology.
[0046] Wherein, when the flow collection unit 110 collects data ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The present invention provides a large-scale data processing system and method, wherein the system includes: a flow collection subsystem and a flow processing subsystem; the flow collection subsystem is used to collect data flow, and mirror the collected data flow , and divide the obtained mirrored traffic into P-way sub-traffic and send it to the traffic storage cluster in the traffic processing subsystem, where P is an integer greater than 1; the traffic storage cluster is composed of M storage servers, and each storage server has Hang N disks, M is a positive integer, N is an integer greater than 1, and M×N≥P; each storage server receives the sub-traffic distributed, and uses load balancing technology to write the sub-traffic distributed to N disks attached to it. In this way, the pressure of continuous writing on the disk is reduced, and the problem of large-scale data storage is better solved.
Description
【Technical field】 [0001] The invention relates to computer network technology, in particular to a large-scale data processing method and system. 【Background technique】 [0002] With the continuous expansion of network users, the amount of data on the Internet has grown explosively, and people have a new understanding of network transmission speed, data security and reliability. User data is widely distributed in many places. For users, data storage and backup that are not well managed pose hidden dangers to business operations. The speed and quality of data transmission affect user experience. In addition, with the development of cloud services Gradually rising and popularizing, the processing needs of large-scale data storage, statistics or analysis have become urgent problems to be solved. However, the existing data processing systems and methods cannot meet the processing requirements of large-scale data due to the impact of performance. For example, if the existing data...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Patents(China)
IPC IPC(8): H04L12/803H04L12/935H04L29/06H04L29/08H04L49/111
Inventor 贺艳军李婷婷周宇石婧岚
Owner BEIJING BAIDU NETCOM SCI & TECH CO LTD