Mass data processing method based on files

Abstract

The invention provides a mass data processing method based on files. The specific processing process comprises the following steps: (1) carrying out exclusive control on multiple data processing servers and keeping only one data processing system to process data and other data processing servers to act as backups; and (2) grouping the files with the mass data according to the total quantity of concurrent threads and ensuring the grouped files to correspond to different threads to be processed. The mass data processing process provided by the invention aims to ensure the correctness and the integrity of the data under the condition that various accidents take place in the computer system and ensure the processability of the mass data while ensuring the correctness and the integrity of the data.

Description

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine

Login to view more

AI Technical Summary

Problems solved by technology

[0003] 1. The amount of data is too large, tens of millions or even hundreds of millions of data. In these data, there may be random data format errors, which will lead to great difficulties in system design;

[0004] 2. The requirements for software and hardware are high. For the processing of massive data, the system resources occupied are high. If the system resources of software and hardware are allocated reasonably, it is also a major problem in the processing of massive data;

[0006] 4. Transaction management of massive data. During data processing, if the data involved is processing a transaction, it is necessary to ensure the transaction control of the data

Images