Big data processing method and big data processing device
What is Al technical title?
Al technical title is built by PatSnap Al team. It summarizes the technical point description of the patent document.
A big data processing and big data technology, applied in the computer field, can solve the problems of reduced user experience, single thread, long running time, etc., to achieve the effect of improving user experience
Inactive Publication Date: 2016-04-13
YONYOU NETWORK TECH
View PDF4 Cites 9 Cited by
Summary
Abstract
Description
Claims
Application Information
AI Technical Summary
This helps you quickly interpret patents by identifying the three key elements:
Problems solved by technology
Method used
Benefits of technology
Problems solved by technology
[0002] At present, the traditional big data processing method is single-server and single-threaded during the running of the program. At this time, if the amount of data is large, it will not only cause a long running time, but also may cause memory overflow.
For example, there are 10 million data to be calculated, and now there are 7 servers in a cluster, namely A, B, C, D, E, F and G, such as figure 1 As shown, when processing, call server E through load balancing, and directly execute 10 million data on a single machine on server E, which not only slows down the calculation efficiency, but also may cause memory overflow
[0003] Moreover, even if a single server is running with multiple threads, when the amount of data is large, running multiple threads may cause threads to wait and the calculation speed of the program to be slow, which greatly reduces the user experience.
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more
Image
Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
Click on the blue label to locate the original text in one second.
Reading with bidirectional positioning of images and text.
Smart Image
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0034] In order to understand the above-mentioned purpose, features and advantages of the present invention more clearly, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the case of no conflict, the embodiments of the present application and the features in the embodiments can be combined with each other.
[0035] In the following description, many specific details are set forth in order to fully understand the present invention. However, the present invention can also be implemented in other ways different from those described here. Therefore, the protection scope of the present invention is not limited by the specific details disclosed below. EXAMPLE LIMITATIONS.
[0036] figure 2 A schematic flowchart of a big data processing method according to an embodiment of the present invention is shown.
[0037] Such as figure 2 As shown, the big data processing me...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more
PUM
Login to view more
Abstract
The invention provides a big data processing method and a big data processing device. The big data processing method comprises the following steps: obtaining a first performance ratio among all servers in a cluster; according to the first performance ratio, selecting a plurality of target servers used for processing a current big data task in all servers according to the first performance ratio; according to a second performance ratio among a plurality of selected target servers, segmenting the current big data task to obtain a plurality of data task blocks; correspondingly distributing the plurality of data task blocks to the plurality of target servers to be independently processed; and receiving a plurality of processing results obtained by that the plurality of target servers correspondingly process the plurality of data task blocks to carry out summary output. Through the technical scheme of the big data processing method, in a multi-thread cluster, the computation efficiency of the big data can be effectively improved, and the problem of memory overflow is solved so as to improve user experience.
Description
technical field [0001] The present invention relates to the field of computer technology, in particular, to a big data processing method and a big data processing device. Background technique [0002] At present, the traditional big data processing method is single-server and single-threaded during the running of the program. At this time, if the amount of data is large, it will not only cause a long running time, but also may cause memory overflow. For example, there are 10 million data to be calculated, and now there are 7 servers in a cluster, namely A, B, C, D, E, F and G, such as figure 1 As shown, when processing, call server E through load balancing, and directly execute 10 million data on a single machine on server E, which not only slows down the calculation efficiency, but also may cause memory overflow. [0003] Moreover, even if a single server is running with multiple threads, when the amount of data is large, running multiple threads may cause threads to wait ...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more
Application Information
Patent Timeline
Application Date:The date an application was filed.
Publication Date:The date a patent or application was officially published.
First Publication Date:The earliest publication date of a patent with the same application number.
Issue Date:Publication date of the patent grant document.
PCT Entry Date:The Entry date of PCT National Phase.
Estimated Expiry Date:The statutory expiry date of a patent right according to the Patent Law, and it is the longest term of protection that the patent right can achieve without the termination of the patent right due to other reasons(Term extension factor has been taken into account ).
Invalid Date:Actual expiry date is based on effective date or publication date of legal transaction data of invalid patent.
Login to view more 
Login to view more