Storage system for mass of real-time data

Abstract

The invention relates to a storage system for a mass of real-time data; the storage system comprises a data query server, data storage servers, data distribution servers, data gateways and terminals, wherein the data query server is respectively connected with the data storage servers and the data distribution servers; the data storage servers are connected with the data distribution servers; and the data distribution servers are connected with the data gateways which are connected with the terminals. Compared with the prior art, the storage system has the advantages that the multiple data storage servers can be configured, i.e. the more terminals can be supported, and the like.

Description

technical field [0001] The invention relates to a data storage system, in particular to a mass real-time data storage system. Background technique [0002] Our system needs to collect a lot of real-time data. Taking the GPS monitoring project of Shandong Linyi Bus Station as an example, each GPS terminal will report real-time data to the data center every 1 minute. The data center needs to record the last report status of each terminal in time, and record all historical records for query. [0003] The current existing solution is: after the data gateway receives a piece of reported data, it will perform an INSERT INTO operation to the database, and every minute, it will perform a large-scale cyclic UPDATE operation to the database to refresh the latest online status record table. [0004] There are some problems with the performance of the current solution. At most, about 700 terminals are online, and the database server has a high CPU usage, which should not be able to co...

AI Technical Summary

Problems solved by technology

This strategy can effectively increase the query speed when querying, but there is a problem when inserting: the implementation of the original solution is: when inserting data, it is implemented through a stored procedure. Before inserting, it is necessary to determine whether there is a target data table. If not exists, generate the target data table

This method is not suitable for frequent insertion, because it needs to be judged every time

In the actual test, it is found that concurrency errors will occur if the number of concurrency exceeds 800. Modifying some of the codes can solve the concurrency errors, but every time you have to make more judgments about whether the target table exists, which will affect the efficiency

[0008] When inserting data, the original solution adopts the method of inserting one by one. This method is not efficient when inserting frequently.

Generally speaking, when SQL Server inserts data into the database in a cyclic manner, 100 records per second has almost reached the limit. Even considering the server factor, the relational database cannot cope with such intensive data insertion operations by conventional methods.

[0009] Refreshing the online status table requires an UPDATE of the latest status of all vehicles every time, and the frequency of this operation is extremely high, which greatly increases the pressure on the database

Images