Mass data processing method based on files

A technology for massive data and data processing, applied in electrical digital data processing, special data processing applications, instruments, etc., can solve problems such as large data volume, low fault tolerance, difficulty, etc., to ensure data security and efficiency. , to ensure the effect of stability

Active Publication Date: 2010-12-15
WUHAN TIANYU INFORMATION IND
View PDF3 Cites 22 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] 1. The amount of data is too large, tens of millions or even hundreds of millions of data. In these data, there may be random data format errors, which will lead to great difficulties in system design;
[0004] 2. The requirements for software and hardware are high. For the processing of massive data, the system resources occupied are high. If the system resources of software and hardware are allocated reasonably, it is also a major problem in the processing of massive data;
[0006] 4. Transaction management of massive data. During data processing, if the data involved is processing a transaction, it is necessary to ensure the transaction control of the data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Mass data processing method based on files
  • Mass data processing method based on files
  • Mass data processing method based on files

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0025] In order to better understand the present invention, the invention will be described in detail below with reference to the drawings and specific embodiments.

[0026] A method for processing massive file-based data of the present invention is to group the massive data received by the data processing system into files first, and then process them concurrently through multiple threads (processes). Such as figure 1 As shown, the data processing system of this embodiment includes a hardware environment consisting of two data processing computers and a disk cabinet. The disk provides data processing computer sharing, and both computers can access the disk cabinet; in addition, the data processing computers are connected to the database server. Connected.

[0027] Such as figure 2 As shown, the specific processing process of this implementation is as follows:

[0028] (1) Mutual exclusive control between multiple data processing servers, the specific steps are as follows:

[0029]...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a mass data processing method based on files. The specific processing process comprises the following steps: (1) carrying out exclusive control on multiple data processing servers and keeping only one data processing system to process data and other data processing servers to act as backups; and (2) grouping the files with the mass data according to the total quantity of concurrent threads and ensuring the grouped files to correspond to different threads to be processed. The mass data processing process provided by the invention aims to ensure the correctness and the integrity of the data under the condition that various accidents take place in the computer system and ensure the processability of the mass data while ensuring the correctness and the integrity of the data.

Description

【Technical field】 [0001] The invention relates to a massive data processing method, in particular to a file-based massive data processing method. 【Background technique】 [0002] Massive data is too large, the data format is complex, and there are many random situations in the data, which is not easy to classify and process, and its processing is a difficult and complicated task. There are mainly the following reasons [0003] 1. The amount of data is too large, tens of millions or even hundreds of millions of data. In these data, there may be random data format errors, which will lead to great difficulties in system design; [0004] 2. The requirements for software and hardware are high. For the processing of massive data, the system resources occupied are high. If the system resources of software and hardware are allocated reasonably, it is also a major problem in the processing of massive data; [0005] 3. High system fault tolerance is required. When an error occurs in ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
Inventor 袁洁
Owner WUHAN TIANYU INFORMATION IND
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap