Mass data processing method based on files

Abstract

The invention provides a mass data processing method based on files. The specific processing process comprises the following steps: (1) carrying out exclusive control on multiple data processing servers and keeping only one data processing system to process data and other data processing servers to act as backups; and (2) grouping the files with the mass data according to the total quantity of concurrent threads and ensuring the grouped files to correspond to different threads to be processed. The mass data processing process provided by the invention aims to ensure the correctness and the integrity of the data under the condition that various accidents take place in the computer system and ensure the processability of the mass data while ensuring the correctness and the integrity of the data.

Description

【Technical field】 [0001] The invention relates to a massive data processing method, in particular to a file-based massive data processing method. 【Background technique】 [0002] Massive data is too large, the data format is complex, and there are many random situations in the data, which is not easy to classify and process, and its processing is a difficult and complicated task. There are mainly the following reasons [0003] 1. The amount of data is too large, tens of millions or even hundreds of millions of data. In these data, there may be random data format errors, which will lead to great difficulties in system design; [0004] 2. The requirements for software and hardware are high. For the processing of massive data, the system resources occupied are high. If the system resources of software and hardware are allocated reasonably, it is also a major problem in the processing of massive data; [0005] 3. High system fault tolerance is required. When an error occurs in ...

AI Technical Summary

Problems solved by technology

[0003] 1. The amount of data is too large, tens of millions or even hundreds of millions of data. In these data, there may be random data format errors, which will lead to great difficulties in system design;

[0004] 2. The requirements for software and hardware are high. For the processing of massive data, the system resources occupied are high. If the system resources of software and hardware are allocated reasonably, it is also a major problem in the processing of massive data;

[0006] 4. Transaction management of massive data. During data processing, if the data involved is processing a transaction, it is necessary to ensure the transaction control of the database. As the amount of data increases, it is necessary to ensure that a large amount of data is processed in the same database transaction. a rather difficult question

[0007] 5. The processing program of massive data, once designed, cannot be reused, and often can only be applied to one industry or a certain project, wasting a lot of manpower and material resources

[0008] In traditional mass data processing, after using high-configuration servers and enhancing CPU processing performance and memory capacity, there are still some problems that cannot be solved, such as low fault tolerance, unreasonable resource allocation, inconsistent transaction management, etc.

Images