Social insurance big data distributed preprocessing method and system

Abstract

The invention discloses a social insurance big data distributed preprocessing method and system. According to the main technical scheme, the method comprises the steps of defining a data preprocessing process as a data preprocessing operation that contains a plurality of preprocessing operation nodes, and concurrently executing the preprocessing operation nodes in independent threads; allocating a plurality of executive threads to a data operation node with high complexity, and concurrently executing the data preprocessing operation by a distributed cloud server cluster; and loading and writing data of the distributed preprocessing system in a distributed file system by column, and performing cache optimization on the data writing operation by utilizing NoSQL. According to the method and the system, the processing performance of preprocessing cloud servers is brought into full play, the performance bottleneck of a single server is overcome, redundant data transmission between the servers and data nodes of the HDFS (Hadoop Distributed File System) is avoided, and the efficiency of loading the data in the HDFS is improved, so that the big data preprocessing efficiency is enhanced.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a method and system for distributed preprocessing of social security big data. Background technique [0002] The national-level informatization planning program "Golden Insurance Project" proposes to comprehensively promote the construction of e-government projects, and promote the informatization of national economic and social development in the way of "government first". Today, social security has covered more than one billion people, and the government has a large amount of social security big data. If the rapid development of big data technology can be used to mine and statistically analyze data in various social security business fields, it can provide a reference for the formulation of government policies and guidelines, realize innovative government services, and further promote the construction of e-government projects. [0003] The mining and analysis of so...

AI Technical Summary

Problems solved by technology

After the data record completes the preprocessing operation node, it arrives at the data loading operation node. However, since the data preprocessing operation and the data loading operation are carried out in two threads at the same time, the time when the data record arrives at the data loading node is inconsistent, which will lead to frequent distributed Cluster file system (HDFS) read and write operations, thus becoming another factor affecting the efficiency of data preprocessing

Images