Distributed data processing system and method

Abstract

The invention provides a distributed data processing system and method. The system comprises a plurality of single-serial servers and further comprises a data collection module for signal data collection, a data storage module for storing complied data information, a data generation module for performing data structure compiling on the data information, a data mining module for scheduling the data information according to the calculated task requirement and a sending module for sending status of the data information meeting the conditions to a monitoring host. The collected device data are preprocessed, the collected mass data are stored in a distributed mode, and the problems that existing data access modes are independent and no dependency or precedence relationship exists are solved; during data mining, a plurality of data scheduling nodes are processed at the same time, the response speed is high, and the data can be sent to the monitoring host within the shortest time; when data is inserted and inquired at the same time, the system performance is not lowered, and the input cost is lowered.

Description

technical field [0001] The present invention mainly relates to the field of data information processing, in particular to a distributed data processing system and method. Background technique [0002] The traditional technical solution uses a relational database, a single database, and a single table for business storage. In terms of technology itself, this method has limited support for data. When the data reaches a certain amount, that is, a mysql single table is greater than 500W records, it will There will be a sharp decline in performance, and it cannot provide full support for the massive data in this system; if it is changed to multi-database and multi-table, it will only solve the storage problem, but the performance will drop a lot, and the complexity of the program will increase suddenly, which will affect the system. The stability will also be reduced, and the conditions for production and on-line will not be met; when the database is inserting a large amount of d...

AI Technical Summary

Problems solved by technology

[0002] The traditional technical solution uses a relational database, a single database, and a single table for business storage. In terms of technology itself, this method has limited support for data. When the data reaches a certain amount, that is, a mysql single table is greater than 500W records, it will There will be a sharp decline in performance, and it cannot provide full support for the massive data in this system; if it is changed to multi-database and multi-table, it will only solve the storage problem, but the performance will drop a lot, and the complexity of the program will increase suddenly, which will affect the system. The stability will also be reduced, and the conditions for production and on-line will not be met; when the database is inserting a large amount of data, simultaneous queries will only lower the two performances, which will directly affect the operation of the business in serious cases, and the accuracy of the data cannot be guaranteed , until the system crashes

Images