Data real-time query system in multi-concurrency and multi-channel environment

Abstract

The invention provides a data real-time query system in a multi-concurrency and multi-channel environment. The data real-time query system comprises a client module, a server module, a database module, a real-time data query module, a Redis module, a timing management module, a data producer module and a cache cleaning module, the system adopts a data caching mechanism to support real-time quick query of data and support that the real-time data query connected with each client is independent in a multi-concurrent environment, the data of different data channels is independent under each clientconnection, the plurality of client connections can share the same data channel, and the real-time data queries are independent from each other and do not influence each other; meanwhile, it is guaranteed that data inquired by each data channel of each client is continuously and progressively increased, and overlapping and omission are avoided; the system can also dynamically set the data cachingtime according to the query frequency so as to effectively utilize memory resources.

Description

Technical field [0001] The invention relates to a real-time data query system in a multi-concurrent multi-channel environment. Background technique [0002] At present, streaming data is widely used. Streaming data is a set of sequential, large, fast, and continuous data sequences. In general, a data stream can be regarded as a dynamic data set that grows indefinitely over time. It is used in the fields of network monitoring, sensor network, aerospace, meteorological measurement and control, and financial services. Streaming data is data that is continuously generated by the data source, and is usually also sent in the form of data records at the same time. Streaming data includes a variety of data, such as log files generated by customers using mobile or web applications, online shopping data, in-game player activity, social networking site information, financial transaction halls or geospatial services, and from connected devices or instruments in the data center Telemetry da...

AI Technical Summary

Problems solved by technology

Since the push is performed sequentially in time, for each client, the real-time data query is not completely concurrent, and when a network link failure occurs on one of the clients, it will cause other clients to display data Delay. In addition, the background needs to manage the subscription and main push logic of the client, and maintain the mapping relationship between data and the client. Once a logic error occurs, it will affect the overall situation;

[0009] (2) The data of multiple channels in the client is superimposed together, and each channel cannot query data independently and concurrently:

[0012] When the client starts to subscribe to real-time data query, there may be unprocessed messages in the message queue. At this time, the message processing module will push these messages to the corresponding channel of the client, resulting in data redundancy. The data received by the client will be missed due to a timeout or other faults

[0013] (4) The cached data does not support the validity period setting, and cannot guarantee that it can be queried repeatedly within a period of time:

[0016] (1) When data is queried in real time in a multi-concurrent and multi-channel environment, the data query between multiple clients is not completely independent, and the background needs to maintain the client's registration subscription and main push management, which is cumbersome;

[0017] (2) When data is queried in real time in a multi-concurrent and multi-channel environment, each client cannot independently query the data of a single channel. When multiple clients share the same channel data, it is unreliable and affects each other.

[0018] (3) When data is queried in real time in a multi-concurrent and multi-channel environment, there may be redundancy and omissions in the data queried for each channel;

[0019] (4) When data is queried in real time in a multi-concurrent and multi-channel environment, dynamic setting of the data cache time is not supported, and the data in the recent period cannot be guaranteed to be repeated and quickly queried

Images