Data processing method, device, equipment and medium based on big data platform
A big data platform and data processing technology, which is applied in the computer field, can solve the problems of long time consumption, increased time, and excessive time consumption, so as to improve reading efficiency, reduce data reading time, and reduce data volume. Effect
Active Publication Date: 2019-02-12
CHINA PING AN LIFE INSURANCE CO LTD
View PDF4 Cites 4 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0002] In the calculation process of big data, every time the data is read, it needs to be read through the hard disk I / O of the machine where the data is stored, and it takes a long time to read the data in the I / O port , if it is necessary to read some data repeatedly, the time consumed for reading at the I / O port will be more
Therefore, the time in the calculation process of big data is increased, and the calculation efficiency is not high
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.
[0028] The data processing method based on the big data platform provided by this application can be applied in such as figure 1 In the application environment, that is, the data processing method is applied in the big data platform, and the big data platform refers to the platform used to store the data of each database in the server cluster and perform operations such as calculation, analysis or processing on the data. The big data platform includes ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a data processing method, device, equipment and medium based on a big data platform. The method comprises the following steps of: acquiring the data table information and the spare memory capacity of each node in the big data platform, wherein the data table information comprises the data table capacity and the data table reading times; sorting the data table according to the order of the reading times of the data table from large to small, and selecting the data table according to the order of sorting according to the spare memory capacity of each node to obtain the data to be cut, wherein the data to be cut is less than or equal to the total spare memory capacity of each node; according to the spare memory capacity of each node, dividing the data to be divided into sections to obtain the partition data; getting the spare capacity ratio of each node's spare memory capacity, and sending partition data to each node's memory according to the spare capacity ratio.The method can reduce the data reading time on the big data platform and improve the data reading efficiency.
Description
technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method, device, equipment and medium based on a big data platform. Background technique [0002] In the calculation process of big data, every time the data is read, it needs to be read through the hard disk I / O of the machine where the data is stored, and it takes a long time to read the data in the I / O port , if it is necessary to read some data repeatedly, the time consumed for reading at the I / O port will be more. Therefore, the time in the calculation process of big data is increased, and the calculation efficiency is not high. Contents of the invention [0003] Based on this, it is necessary to address the above technical problems and provide a data processing method, device, computer equipment and storage medium based on a big data platform that can reduce data reading time and improve data reading efficiency. [0004] A data processing meth...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/21
Inventor 王海平
Owner CHINA PING AN LIFE INSURANCE CO LTD