A method, system, electronic device and storage medium for landing and analyzing large quantities of time-sensitive data based on a time series database

Abstract

The present invention discloses a method, system, electronic device and storage medium for landing and analyzing large quantities of time-sensitive data based on a time series database, adopts data acquisition technology to collect data from network equipment, and adjusts the parameters of the acquisition time interval and data type in the data acquisition through a configuration file in the process; distributes data messages to different message queues according to different data types and device addresses; stores the data in each message queue in batches through partitions and tables according to the acquisition time, data type and device address to achieve data landing; the acquisition time includes year, month, day, hour, minute and second; sets query parameters, including: data type, belonging device and acquisition time, and obtains the query results required by the business through retrieval and analysis of the query parameters. The present invention makes the acquisition and use of data more flexible and fast, and greatly improves the efficiency and utilization of data acquisition.

Description

Technical Field

[0001] The present invention relates to the field of network operation and maintenance technology, and in particular to a method, system, electronic device and storage medium for landing and analyzing large quantities of time-sensitive data based on a time series database. Background Art

[0002] In the field of operator network communications, the operating status of equipment is an important indicator for judging the health of equipment. With the popularization of networks and the emergence of new technologies, the scale of networks is increasing, the complexity of deployment is gradually increasing, and users' requirements for service quality are also increasing. In order to meet user needs, network operation and maintenance must be more refined and intelligent. Today's network operation and maintenance faces the following challenges: 1) Super large scale: There are a large number of managed devices and a large amount of monitored information; 2) Fast positioning: In a complex network, faults can be quickly located, reaching a fault positioning speed of seconds or even sub-seconds; 3) Fine monitoring: More types of monitored data and finer monitoring granularity, so as to fully and accurately reflect the network status, estimate possible faults based on this, and provide a strong data basis for network optimization. The network operation and maintenance system needs to monitor not only the traffic statistics on the interface, the packet loss on each flow, the CPU and memory usage, but also the delay jitter of each flow, the delay of each message on the transmission path, and the buffer occupancy on each device.

[0003] Traditional network monitoring methods (SNMP, CLI, logs) can no longer meet network needs. SNMP and CLI mainly use the "pull mode" to obtain data, that is, sending requests to obtain data on the device, which limits the number of network devices that can be monitored and cannot quickly obtain data; although SNMP Trap and logs use the "push mode" to obtain data, that is, the device actively reports data to the monitoring device, but only reports events and alarms. The monitored data content is extremely limited and cannot accurately reflect the network status; Telemetry is a remote data collection technology that monitors device performance and faults. The "push mode" is used to obtain rich monitoring data in a timely manner, which can realize the rapid location of network faults, thereby solving the above-mentioned network operation and maintenance. While Telemetry meets the real-time collection requirements, such a large amount of data storage and data query has become a problem.

[0004] The controller needs to collect and store various performance data of the managed network devices in real time, such as traffic data, CPU utilization, memory utilization, transmit and receive data, SRV6 data, etc., and display these data in real time during the use of the controller business, and perform algorithm analysis on historical data to determine the operating status of the network equipment, so as to judge the equipment situation. This requires the controller itself to have the ability to collect equipment performance data in seconds, and store these data flexibly in seconds, minutes, hours, days, weeks, months, years and other time units according to the business. The daily amount of new data and archived data exceeds 10 billion according to the existing business status. The time interval for collecting equipment performance data needs to be based on the dynamic changes of the business, the amount of collected data is huge, and the data type is diverse. Therefore, the system's use of performance data has the problem of difficulty in adjusting the collection interval. Data storage needs to be dynamically adjusted according to the collected data type, data level, and data timeliness, resulting in problems such as squeezed consumption of collected data and slow query. Summary of the invention

[0005] In view of the above-mentioned situation, the purpose of the present invention is to provide a method, system, electronic device and storage medium for the implementation and analysis of large quantities of time-sensitive data based on a time series database, so as to solve the problems arising from data collection and subsequent storage and use, and make data collection and use more flexible and rapid.

[0006] To achieve the above purpose, the technical solution provided by the present invention is:

[0007] A method for landing and analyzing large quantities of time-sensitive data based on a time series database, adopts data acquisition technology to collect data from network devices, and in this process adjusts the parameters of the collection time interval and data type in the data collection through a configuration file; distributes the collected data messages to different message queues according to different data types and device addresses; stores the data in each message queue in batches through partitions and tables according to the data collection time, data type and device address, so as to realize data landing; the collection time includes year, month, day, hour, minute and second; sets query parameters for querying the stored data, and the setting of the query parameters corresponds to the calculation process of partitions and tables, including: data type, belonging device and collection time, and obtains the query results required by the business through retrieval and analysis of the query parameters.

[0008] In order to optimize the above technical solutions, the specific measures / limitations taken also include:

[0009] The data collection technology used to collect data from network devices is specifically the use of telemetry technology to collect data from network devices.

[0010] The data in each message queue is stored in batches by partitioning and tabulating according to the data collection time, data type and device address. The specific steps are as follows:

[0011] (1) The partition table space where the data is located is located by performing hash calculation on the device to which the data belongs;

[0012] (2) After locating the partition table space, determine which type of table the data belongs to based on the data collection type, and locate the data to the type attribution table in the partition table space;

[0013] (3) After locating the type attribution table, the physical attribution of the data is determined by the year, month, and day of data collection, and the data is located in the physical table in the type attribution table, which includes a year table, a month table, and a day table;

[0014] (4) After locating the physical table, create a time series table and store the data in the time series table according to the hour, minute, and second of data collection.

[0015] Furthermore, the time interval for data storage and the amount of data stored are configured in the timing table to selectively store data.

[0016] Furthermore, the data in the time series table that exceeds the time limit is cleared in a timely manner.

[0017] The present invention also protects a system for landing and analyzing large quantities of time-sensitive data based on a time series database, including:

[0018] The data collection module is used to collect data from network devices using data collection technology, and in this process, the parameters of the collection time interval and data type in the data collection are adjusted through the configuration file;

[0019] The message module is used to distribute the collected data messages to different message queues according to different data types and device addresses;

[0020] A timing algorithm module, used to determine the partition and table of data in each message queue according to the data collection time, data type and device address, wherein the collection time includes year, month, day, hour, minute and second;

[0021] A data storage module is used to store data in batches according to the calculated partition and table results;

[0022] The data query template is used to query the stored data. The query parameters set correspond to the calculation process of partitioning and table division, including: data type, belonging device and collection time. The query results required by the business are obtained by retrieving and analyzing the query parameters.

[0023] Furthermore, the partition table includes a partition table space calculated according to the data attribution device, a type attribution table calculated according to the data collection type, a physical table calculated according to the year, month and day of data collection, and a timing table calculated according to the hour, minute and second of data collection.

[0024] The present invention also protects an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for landing and analyzing large quantities of time-sensitive data based on a time series database as described above is implemented.

[0025] The present invention also protects a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for landing and analyzing a large amount of time-sensitive data based on a time series database as described above.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The method of the present invention can automatically calculate the partition table, which not only satisfies the requirement that the system can automatically identify and make new storage adjustments when the data type and magnitude change, but also can make new data compatible without updating the system.

[0028] The present invention supports automatic horizontal and vertical expansion, which can not only perform traditional node expansion, but also configure vertical expansion for new data types and new message queues;

[0029] The time series partition table algorithm proposed in the present invention can automatically select the use of the time series partition table, making the data distribution more uniform and avoiding the uncertainty of batch data causing extreme data to tilt to a certain partition table;

[0030] The present invention can realize indifferent storage of unlimited data types, and can perform non-perceptual storage of different data messages. In addition, different types of aggregate queries can be performed on data during query. When storing different types of data, the system converts them into JSON for automatic storage. There is no need to consider the data structure of new data types, and only JSON needs to be processed during data query.

[0031] The present invention enables the system to support the storage and query of tens of billions of data at a relatively low configuration. The main implementation method is to divide the data into blocks, partitions and tables for storage and query according to the characteristics of data type, collection time and timeliness.

[0032] The present invention is highly flexible. From data collection to information consumption to data landing, all dynamic information supports configurable adjustment, which is more flexible. Through flexible collection configuration, the diversity and timeliness of data collection can be achieved. The collection efficiency and query speed of backbone network equipment can be improved, which is of great help to the real-time monitoring and early warning of the operation status of the equipment.

[0033] The present invention can be applied to the collection and use of device performance data, which greatly improves the efficiency and utilization of data collection; the present invention can be applied to the equipment information collection and storage query of various networks of operators; based on the time series algorithm template of the present invention, it can be applied to various scenarios using time series databases for large-scale data storage, so that in the use of time-based data storage needs, different types of time data storage become more recommended and efficient, reducing the development workload caused by the addition of new data types by developers and reducing operation and maintenance costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : A flow chart of the method for implementing and analyzing large quantities of time-sensitive data based on a time series database of the present invention. DETAILED DESCRIPTION

[0035] The above contents of the present invention are further described in detail below in the form of embodiments, but this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the above contents of the present invention belong to the scope of the present invention.

[0036] Some of the terms in the present invention are explained as follows:

[0037] The full name of the time series database is time series database. The time series database is mainly used to process data with time tags (changing in the order of time, that is, time serialization). Data with time tags is also called time series data.

[0038] Time series data are mainly collected and generated by various types of real-time monitoring, inspection and analysis equipment in the power industry, chemical industry, meteorological industry, geographic information, etc. The typical characteristics of these industrial data are: fast generation frequency (each monitoring point can generate multiple data within one second), heavy reliance on collection time (each data is required to correspond to a unique time), large number of measuring points and large amount of information (conventional real-time monitoring systems have thousands of monitoring points, which generate data every second and tens of GB of data every day).

[0039] Telemetry, also known as Network Telemetry, is a technology that collects data remotely and at high speed from physical or virtual devices. The device periodically and actively sends information such as interface traffic statistics, CPU or memory data of the device to the collector through push mode. Compared with the traditional question-and-answer interaction of pull mode, it provides more real-time and high-speed data collection function.

[0040] In one embodiment, the present invention proposes a method for landing and analyzing large quantities of time-sensitive data based on a time series database, and uses data collection technology to collect data from network devices. In this process, the parameters of the collection time interval and data type in the data collection are adjusted through a configuration file; the collected data is distributed to different message queues according to different data types and device addresses; the data in each message queue is stored in batches through partitions and tables according to the data collection time, data type and device address to achieve data landing; the collection time includes year, month, day, hour, minute and second; query parameters are set for querying the stored data, and the setting of the query parameters corresponds to the calculation process of the partition and table, including: data type, belonging device and collection time, and the query results required by the business are obtained by retrieving and analyzing the query parameters.

[0041] Data collection technology is used to collect data from network devices, specifically, telemetry technology is used to collect data from network devices.

[0042] The data collection template can realize the second-level collection of network devices through telemetry technology, and the present invention adds the configuration of collection time interval and collection type on this basis. R&D and operation and maintenance personnel can adjust these two parameters through the configuration file.

[0043] The message template distributes the data collected by the data collection template to different message queues according to different data types and device addresses, so as to avoid the message being pushed to a certain message queue, resulting in a backlog of message queue data.

[0044] The time series algorithm is the core module of the present invention. The time series algorithm uses the year, month, day, hour, minute, second, data type, device and content of the data collection time as input parameters to determine the partition and table where the data is located.

[0045] According to the data collection time, data type and device address, the data in each message queue is stored in batches through partitions and tables. The specific steps are as follows:

[0046] (1) By performing hash calculation on the device to which the data belongs, the partition table space where the data is located is located, so that the number of data devices in each table space is evenly distributed;

[0047] (2) After locating the partition table space, determine which type of table the data belongs to based on the data collection type, and locate the data to the type attribution table in the partition table space, such as the flow data belonging to the flow table;

[0048] (3) After locating the type attribution table, determine the physical attribution of the data by the year, month, and day of data collection, and locate the data to the physical table in the type attribution table. The physical table includes the year table, the month table, and the day table.

[0049] (4) After locating the physical table, create a time series table and store the data in the time series table according to the hour, minute, and second of data collection. For example, if the system daily table is configured to create a time series table by hour, the system will create a new time series table to store data when it reaches the hour.

[0050] Configure the time interval for data storage and the amount of data stored in the time series table to selectively store data.

[0051] This time series algorithm not only calculates the data landing table, but also selectively stores physical tables of different times according to the content of the data and the collection time. For example, if the year table is configured to store a piece of data for one hour, the algorithm will take the modulus based on the hour of the collection time and filter and store a piece of data that satisfies the whole hour minus the number of seconds of the collection time interval.

[0052] Clear out the data in the time series table that exceeds the validity period in a timely manner.

[0053] In another embodiment, the present invention proposes a system for landing and analyzing large quantities of time-sensitive data based on a time series database, including:

[0054] The data collection module is used to collect data from network devices using data collection technology, and in this process, the parameters of the collection time interval and data type in the data collection are adjusted through the configuration file;

[0055] The message module is used to distribute the collected data messages to different message queues according to different data types and device addresses;

[0056] The timing algorithm module is used to determine the partition and table of the data in each message queue according to the data collection time, data type and device address. The collection time includes year, month, day, hour, minute and second;

[0057] A data storage module is used to store data in batches according to the calculated partition and table results;

[0058] The data query template is used to query the stored data. The query parameters set correspond to the calculation process of partitioning and table division, including: data type, belonging device and collection time. The query results required by the business are obtained by retrieving and analyzing the query parameters.

[0059] Furthermore, the partition table includes a partition table space calculated according to the data to which the data belongs, a type attribution table calculated according to the data collection type, a physical table calculated according to the year, month and day of the data collection, and a timing table calculated according to the hour, minute and second of the data collection.

[0060] In another embodiment, the present invention proposes an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for landing and analyzing large quantities of time-sensitive data based on a time series database is implemented as described above.

[0061] In another embodiment, the present invention proposes a computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the above method for landing and analyzing a large amount of time-sensitive data based on a time series database.

[0062] The present invention is further described in detail below in conjunction with specific embodiments:

[0063] In a data collection scenario, the controller collects traffic information of the device port at a collection dimension of 10 seconds. After the operation and maintenance personnel configure the corresponding data type and collection interval in the data collection module, Telemetry collects the device traffic information.

[0064] The message module distributes the collected information to each message queue partition according to device maintenance, and notifies the timing algorithm module for processing.

[0065] The controller uses traffic data mainly to present traffic diagrams of network equipment circuits to network administrators, including real-time traffic information within 1 hour at a 10-second granularity, traffic diagram information within 24 hours at a 5-minute dimension, and annual traffic diagram information at an hourly dimension.

[0066] The operation and maintenance configuration personnel configure the parameters to the timing algorithm module based on these business requirements. When the timing algorithm module processes specific traffic information, it determines the table space for data storage based on the device to which the data belongs, creates a specific physical table based on the data type, and then performs timing calculations based on the parameters.

[0067] Since the business wants 10-second data within an hour, all data is stored in a real-time table (time series table). The business also needs traffic data within 24 hours at 5-minute intervals. The algorithm uses the minute where the data is collected / 5 as an integer and ensures that the second where the data is collected is <10. In this way, a unique valid data is obtained and the data is placed in the set of the 24-hour physical table. Similarly, the data in the year table is obtained.

[0068] The data storage module obtains each table and the data set corresponding to each table to process the data into the warehouse, and clears out data that has exceeded its timeliness, such as data in the real-time table that is older than one hour.

[0069] When a page queries the traffic of a port of a device, the data query service determines the set of tables to which the data belongs based on the query parameters, searches these tables, and returns the data to the user.

[0070] Through this algorithm, each piece of data can be classified and stored according to business requirements, and deleted after the data expires. This greatly saves the space required for data storage; it also makes the data in each table evenly distributed, and the real-time adjustment of table data also reduces the amount of data in a single physical table, greatly improving query efficiency.

[0071] In the embodiments disclosed in the present application, the computer storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. The computer storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of computer storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0072] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0073] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any technician familiar with the profession, without departing from the scope of the technical solution of the present invention, according to the technical essence of the present invention, any simple modification, equivalent replacement and improvement made to the above embodiment still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for landing and analyzing large quantities of time-sensitive data based on a time series database, characterized by: Data collection technology is used to collect data from network devices. In this process, the parameters of the collection time interval and data type in the data collection are adjusted through the configuration file; the collected data are distributed to different message queues according to different data types and device addresses; the data in each message queue is stored in batches through partitions and tables according to the data collection time, data type and device address to realize data landing; the collection time includes year, month, day, hour, minute and second; Setting query parameters for querying the stored data. The setting of query parameters corresponds to the calculation process of partitioning and table division, including: data type, belonging device and collection time. The query results required by the business are obtained by retrieving and analyzing the query parameters; The data in each message queue is stored in batches by partitioning and tabulating according to the data collection time, data type and device address. The specific steps are as follows: (1) By performing hash calculation on the device to which the data belongs, the partition table space where the data is located is located; (2) After locating the partition table space, determine which type of table the data belongs to based on the data collection type, and locate the data to the type attribution table in the partition table space; (3) After locating the type attribution table, determine the physical attribution of the data by the year, month, and day of data collection, and locate the data to the physical table in the type attribution table, which includes the year table, the month table, and the day table; After locating the physical table, create a time series table and store the data in the time series table according to the hour, minute, and second of the data collection.

2. According to claim 1, the method for landing and analyzing large quantities of time-sensitive data based on a time series database is characterized in that: The data collection technology used to collect data from network devices is specifically the use of telemetry technology to collect data from network devices.

3. The method for landing and analyzing large quantities of time-sensitive data based on a time series database according to claim 1 is characterized in that: The time interval for data storage and the amount of data stored are configured in the timing table to selectively store data.

4. According to claim 1, the method for landing and analyzing large quantities of time-sensitive data based on a time series database is characterized in that: Clear out the data in the time series table that exceeds the validity period in a timely manner.

5. The system used by the method for landing and analyzing large quantities of time-sensitive data based on a time series database as described in claim 1 is characterized in that: include: The data collection module is used to collect data from network devices using data collection technology, and in this process, the parameters of the collection time interval and data type in the data collection are adjusted through the configuration file; The message module is used to distribute the collected data messages to different message queues according to different data types and device addresses; A timing algorithm module, used to determine the partition and table of data in each message queue according to the data collection time, data type and device address, wherein the collection time includes year, month, day, hour, minute and second; A data storage module is used to store data in batches according to the calculated partition and table results; The data query template is used to query the stored data. The query parameters set correspond to the calculation process of partitioning and table division, including: data type, belonging device and collection time. The query results required by the business are obtained by retrieving and analyzing the query parameters.

6. The system used by the method for landing and analyzing large quantities of time-sensitive data based on a time series database according to claim 5 is characterized in that: The partition table includes a partition table space calculated according to the data attribution device, a type attribution table calculated according to the data collection type, a physical table calculated according to the year, month and day of data collection, and a time series table calculated according to the hour, minute and second of data collection.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for landing and analyzing large quantities of time-sensitive data based on a time series database as described in any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to execute the method for landing and analyzing large quantities of time-sensitive data based on a time series database as described in any one of claims 1 to 4.

Images