Concurrent time series data loading method and device

Abstract

The invention discloses a concurrent time series data loading method. The method comprises the following steps: obtaining a unique identifier of same type of data in the time series data, presetting a quantity of data processors, and determining the name sequence number of a data processor corresponding to the type of data according to the quantity of data processors and the unique identifier of the same type of data; creating subscripts and a queue array storage space corresponding to the name sequence number of the data processor, and respectively setting a queue depth and an array length; receiving time series data consistent with data types of the data processors by the data processors, analyzing the attributes and values in the obtained time series data, and storing the attributes and the values in a storage position of a subscript corresponding to the queue array; and when a queue in the queue array achieves the set depth, triggering a database interface to obtain all the data in the queue array and carry out loading operation. Through the method disclosed in the invention, the efficient loading of time series data can be realized. The invention furthermore discloses a concurrent time series data loading device.

Description

technical field [0001] The invention relates to the field of data processing, in particular to a technology for sending time series data into a database concurrently. Background technique [0002] The so-called time-series data means that the data is generated in a time-series manner, and its time-sequence needs to be maintained during a series of data operations such as transmission and warehousing, such as alarm data, index data, and performance data in the telecommunications system. [0003] Taking the alarm data as an example, the alarm data must ensure timing during transmission, and the sequence of alarms generated by the same device cannot be disrupted. If the alarm data cannot guarantee timing, it will cause errors in alarm storage and subsequent use. various problems. In the early design, the alarm data is implemented by a single thread or a single process from receiving to processing to storage to ensure its timing. With the continuous expansion of alarm equipmen...

AI Technical Summary

Problems solved by technology

[0005] The ability to process data is extremely low for this single-process technology for processing time-series data into the database

It cannot meet the increasing data volume processing requirements. With the rapid increase of data volume, its defects are becoming more and more obvious. First, the single-thread or single-process processing method results in low efficiency of data storage. When the number of alarms is large, single-thread or single-process Poor process efficiency leads to data accumulation that affects system operation and damages other businesses; secondly, frequent database operations result in frequent DB interaction, which in turn causes continuous I / O consumption of the operating system; due to the occupation of MAP memory, untimely recycling leads to excessive consumption of the operating system , based on the above reasons, when a single process is used to store time-series data, the greater the data volume, the lower the efficiency, which will have a direct or indirect impact on the carrying system

Images