Lots of real time data storage distribution and query information processing method

Abstract

The invention relates to a lots of real time data storage distribution and query information processing method. The method comprises the following steps that: 1). A data query server is started, after the starting, a corresponding table of data gateway corresponding to each terminal is loaded into a memory and a query service is provided externally; 2). A data storage server is started, is used to store data distributed by a data distribution server and simultaneously response the distributed query of the data query server; 3). The data distribution server carries out data distribution; 4). The data query server carries out the query. Compared to the prior art, by using the method of the invention, a plurality of the data storage servers can be configured which means more numbers of terminals can be supported.

Description

technical field [0001] The invention relates to a data storage method, in particular to a massive real-time data storage distribution and query information processing method. Background technique [0002] Our system needs to collect a large amount of real-time data. Taking the GPS monitoring project of Linyi Bus Station in Shandong as an example, each GPS terminal will report real-time data to the data center every 1 minute. The data center needs to record the last report status of each terminal in time, and record all historical records for query. [0003] The current existing solution is: after the data gateway receives a piece of reported data, it will perform an INSERT INTO operation to the database, and every minute, it will perform a large-scale cyclic UPDATE operation to the database to refresh the latest online status record table. [0004] There are some problems with the performance of the current solution. At most, about 700 terminals are online, and the database...

AI Technical Summary

Problems solved by technology

This strategy can effectively increase the query speed when querying, but there is a problem when inserting: the implementation of the original solution is: when inserting data, it is implemented through a stored procedure. Before inserting, it is necessary to determine whether there is a target data table. If not exists, generate the target data table

This method is not suitable for frequent insertion, because it needs to be judged every time

In the actual test, it is found that concurrency errors will occur if the number of concurrency exceeds 800. Modifying some of the codes can solve the concurrency errors, but every time you have to make more judgments about whether the target table exists, which will affect the efficiency

[0008] When inserting data, the original solution adopts the method of inserting one by one. This method is not efficient when inserting frequently.

Generally speaking, when SQL Server inserts data into the database in a cyclic manner, 100 records per second has almost reached the limit. Even considering the server factor, the relational database cannot cope with such intensive data insertion operations by conventional methods.

[0009] Refreshing the online status table requires an UPDATE of the latest status of all vehicles every time, and the frequency of this operation is extremely high, which greatly increases the pressure on the database

Images