Data processing method and device

Abstract

The data processing method and device are applied to front-end equipment in a data processing system, the data processing system further comprises target equipment and a database, the target equipment is used for collecting multiple pieces of time sequence data in the operation process of the target equipment, and each piece of time sequence data comprises collection time of the time sequence data. The method comprises the following steps that: front-end equipment acquires first time sequence data and judges whether the first time sequence data is sequential data or not; further, under the condition that the front-end equipment determines that the first time sequence data is sequential data, the first time sequence data are stored in a database; or, under the condition that the front-end equipment determines that the first time sequence data are out-of-order data, the first time sequence data are stored in the sorting cache of the front-end equipment. In this way, the influence on the database performance caused by the fact that the front-end equipment stores the out-of-order data into the database is avoided.

Description

[0001] This application claims priority to a Chinese patent application filed with the Intellectual Property Office of the People's Republic of China on July 12, 2023, with application number 202310854392.7 and invention name "A method, device and computing device cluster for processing time series data", all contents of which are incorporated by reference in this application. Technical Field

[0002] The embodiments of the present application relate to the field of cloud computing, and in particular to a data processing method and device. Background Art

[0003] Time series data refers to a data column recorded in chronological order. Each data in the same data column can be the operating data of the same indicator and must be comparable. Time series data is very common in today's Internet of Things (IOT), cloud computing operation and maintenance, and financial fields. After being sampled by the sampling end, time series data usually needs to be transmitted to the collection end for unified analysis and processing.

[0004] However, due to factors such as network latency, the order of arrival of the time series data received by the collection end is often inconsistent with the result of sorting according to the collection time of the time series data. This kind of time series data is described as out-of-order data. After the collection end stores multiple time series data in the database, if out-of-order data arrives, the newly arrived out-of-order data cannot be directly stored in the database. Instead, it is necessary to pull up the multiple historical time series data before the out-of-order data in the database first, re-sort the pulled up multiple historical time series data and the out-of-order data before storing them.

[0005] This has a significant impact on the performance of the database. Summary of the invention

[0006] The present application provides a data processing method and device for reducing the impact of disordered data on database performance.

[0007] In a first aspect, the present application provides a data processing method, which is applied to a front-end device of a data processing system. Exemplarily, the method can be executed by the front-end device or a module in the front-end device. For ease of description, the following takes the front-end device executing the method as an example.

[0008] The data processing system also includes a target device and a database. The target device is used to collect multiple time series data during the operation of the target device. Each time series data includes the collection time of the time series data.

[0009] The method includes: the front-end device obtains the first time series data and determines whether the first time series data is sequential data. Further, when the front-end device determines that the first time series data is sequential data, the first time series data is stored in a database; or, when the front-end device determines that the first time series data is out-of-order data, the first time series data is stored in a sorting cache of the front-end device.

[0010] In the above technical solution, after the front-end device obtains the first time series data, it needs to determine whether the first time series data is sequential data. When it is determined that the first time series data is sequential data, the first data will be stored in the database, and when it is determined that the first time series data is out of order data, the first time series data will be stored in the sorting cache of the front-end device. In this way, the front-end device is prevented from storing out of order data in the database. When the acquisition time of the newly received data is earlier than the acquisition time of the time series data already stored in the database, the front-end device reads the time series data in the database again, and sorts the read data and the newly received data, which affects the database performance. That is, this method can reduce the impact on database performance, reduce resource consumption for data processing, and reduce the problem of write amplification.

[0011] In addition, when the front-end device determines that the first time series data is sequential data, it will store the first time series data in the database, without first storing the first time series data in the sorting cache, and sorting the multiple time series data in the sorting cache whose arrival time is within the preset time window before storing them in the database, thereby solving the problem of untimely processing of time series data. That is, it helps to improve the efficiency of data processing.

[0012] In a possible implementation, when the front-end device determines whether the first time series data is sequential data, specifically, the front-end device determines whether the first time series data is the next time series data after the second time series data, and the second time series data is the time series data most recently stored in the database by the front-end device. Further, when the front-end device determines that the first time series data is the next time series data after the second time series data, it determines that the first time series data is sequential data; when the front-end device determines that the first time series data is not the next time series data after the second time series data, it determines that the first time series data is out of order data. As above, a method for the front-end device to determine whether the first time series data is sequential data is provided.

[0013] In a possible implementation, after the front-end device stores the first time series data in the database, it can also obtain the third time series data from the sorting cache of the front-end device, wherein the third time series data is the time series data in the sorting cache that is collected earlier than other time series data, and the front-end device determines whether the third time series data is sequential data, and if it is determined that the third time series data is sequential data, it stores the third time series data in the database. In the above technical solution, once the front-end device stores the first time series data in the database, it can obtain the time series data (i.e., the third time series data) collected earlier than other time series data from the sorting cache, and then after determining that the third time series data is sequential data, write the third time series data into the database, which helps to further improve the efficiency of data processing.

[0014] In a possible implementation, after the front-end device stores the first time series data in the sorting cache of the front-end device, if it is determined that the number of time series data in the sorting cache exceeds a preset number, the time series data collected earlier than other time series data in the sorting cache is stored in the database. In the above technical solution, when the front-end device determines that the number of time series data included in the sorting cache exceeds a preset number, it is determined that a certain time series data collected earlier than all the time series data in the sorting cache has been lost, and then the time series data collected earlier than other time series data in the sorting cache is stored in the database, so as to avoid the problem that the front-end device has been waiting for a certain time series data that has been lost, resulting in a prolonged data processing time, that is, it helps to further improve the efficiency of data processing.

[0015] In a possible implementation, when the front-end device stores the first time series data in the sorting cache of the front-end device, the front-end device may specifically determine the location of the sorting cache to store the first time series data according to the first acquisition time of the first time series data and the acquisition time of the time series data already in the sorting cache; and store the first time series data in the sorting cache according to the location. In the above technical solution, the front-end device sorts the multiple time series data in the sorting cache according to the acquisition time, which helps the front-end device to quickly determine the time series data with the earliest acquisition time from the sorting cache, and helps to further improve the efficiency of data processing.

[0016] In a possible implementation, when the front-end device determines whether the first time series data is the next time series data after the second time series data, specifically, the front-end device determines whether the first time series data is the next time series data after the second time series data based on the reference acquisition time, the acquisition rule and the first acquisition time of the first time series data, wherein the acquisition rule is the rule of the target device collecting time series data pre-learned based on multiple historical time series data, and the reference acquisition time is the second acquisition time of the second time series data. In the above technical solution, the front-end device can obtain the acquisition rule, which is the pre-learned rule of the target device collecting time series data. Furthermore, the front-end device can use the acquisition time of the previous time series data (i.e., the second time series data) stored in the database as the reference acquisition time, and then determine whether the first time series data is the next time series data after the second time series data based on the reference acquisition time, the acquisition rule and the first acquisition time of the first time series data, which helps to improve the accuracy of the judgment.

[0017] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the front-end device determines whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it can be specifically that when the front-end device determines that the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data; when the front-end device determines that the difference between the first collection time and the benchmark collection time is not equal to the fixed time interval, it is determined that the first time series data is not the next time series data after the second time series data. In a possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the mean value and dynamic frequency tolerance of the exponential distribution; when the front-end device determines whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule, and the first collection time of the first time series data, it can be specifically that when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, it is determined that the first time series data is the next time series data of the second time series data; when it is determined that the difference between the first collection time and the benchmark collection time is greater than a threshold, it is determined that the first time series data is not the next time series data of the second time series data, wherein the threshold is determined by the mean value and dynamic frequency tolerance of the exponential distribution. As described above, providing two different collection rules, and an implementation method in which the front-end device determines whether the first time series data is the next time series data of the second time series data based on the collection rule, helps to improve the accuracy of the judgment.

[0018] In a possible implementation, after storing the first time series data in the database, the front-end device also updates the reference acquisition time to the first acquisition time of the first time series data. In the above technical solution, the front-end device can update the reference acquisition time according to the acquisition time of the first time series data most recently stored in the database, and then determine whether the next time series data is sequential data (i.e., the next time series data of the first time series data acquired by the target device) according to the updated reference acquisition time, which helps to improve the accuracy of the judgment.

[0019] In a second aspect, the present application provides a data processing device, which is applied to a front-end device of a data processing system, wherein the data processing system further includes a target device and a database, wherein the target device is used to collect multiple time series data during the operation of the target device, wherein each time series data includes the collection time of the time series data. Exemplarily, the device can be a front-end device or a module in the front-end device.

[0020] The device includes: an acquisition module, used to acquire first time series data; a judgment module, used to judge whether the first time series data is sequential data; a storage module, used to store the first time series data in a database when the judgment module determines that the first time series data is sequential data; or, when the judgment module determines that the first time series data is out of order data, store the first time series data in a sorting cache of a front-end device.

[0021] In a possible implementation, when the judgment module judges whether the first time series data is sequential data, it is specifically used to: judge whether the first time series data is the next time series data after the second time series data, and the second time series data is the time series data most recently stored in the database by the front-end device.

[0022] In a possible implementation, after the storage module stores the first time series data in a database, the acquisition module is further used to: obtain third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache whose collection time is earlier than other time series data; the judgment module is further used to: determine whether the third time series data is sequential data; the storage module is further used to: store the third time series data in the database when the judgment module determines that the third time series data is sequential data.

[0023] In a possible implementation, after storing the first time series data in the sorting cache of the front-end device, the storage module is also used to: when it is determined that the number of time series data in the sorting cache exceeds a preset number, store the time series data in the sorting cache that is collected earlier than other time series data in the database.

[0024] In a possible implementation, when the storage module stores the first time series data in the sorting cache of the front-end device, it is specifically used to: determine the position of storing the first time series data in the sorting cache according to the first acquisition time of the first time series data and the acquisition time of the existing time series data in the sorting cache; and store the first time series data in the sorting cache according to the position.

[0025] In one possible implementation, when the judgment module judges whether the first time series data is the next time series data after the second time series data, it is specifically used to: judge whether the first time series data is the next time series data after the second time series data based on a benchmark collection time, a collection rule and a first collection time of the first time series data, wherein the collection rule is a rule for collecting time series data of a target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

[0026] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the judgment module judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, determine that the first time series data is the next time series data after the second time series data.

[0027] In a possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the average value and dynamic frequency tolerance of the exponential distribution; when the judgment module judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, determine that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value and dynamic frequency tolerance of the exponential distribution.

[0028] In a possible implementation, after storing the first time series data in the database, the judgment module is further configured to: update the reference acquisition time to the first acquisition time of the first time series data.

[0029] In a third aspect, the present application provides a data processing system, comprising: a front-end device, a target device and a database; wherein the target device is used to collect multiple time series data during the operation of the target device, each time series data includes the collection time of the time series data; the front-end device is used to execute the method in the above-mentioned first aspect or any possible implementation method of the first aspect.

[0030] In a fourth aspect, the present application provides a computing device cluster, comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0032] In a sixth aspect, the present application provides a computer-readable storage medium, comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0033] The technical effects that can be achieved in any of the second to sixth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A structural diagram of a data processing system;

[0035] Figure 2 It is a schematic diagram of the hardware structure of a front-end device;

[0036] Figure 3 A flowchart of the first data processing method provided in this application;

[0037] Figure 4 A flowchart of the second data processing method provided in this application;

[0038] Figure 5 A flowchart of the third data processing method provided in this application;

[0039] Figure 6 A flow chart of a learning collection rule provided for this application;

[0040] Figure 7 A schematic diagram of a process for a front-end device provided in this application to determine whether a collection interval is regular;

[0041] Figure 8 A schematic diagram of state switching of a training cache provided in this application;

[0042] Fig. 9 A schematic diagram of another state switching of a training cache provided by the present application;

[0043] Fig.10 A schematic diagram of a learning collection rule under multiple indicators provided in this application;

[0044] Fig.11 A schematic diagram of data sorting under multiple indicators provided in this application;

[0045] Fig.12 A schematic diagram of a fourth data processing method provided in this application;

[0046] Fig.13 A schematic diagram of the structure of a data processing device provided in this application;

[0047] Fig.14 A schematic diagram of the structure of a computing device provided for this application;

[0048] Fig.15 A schematic diagram of the structure of a computing device cluster provided for this application;

[0049] Fig.16 A schematic diagram of network connections in a computing device cluster provided in this application. DETAILED DESCRIPTION

[0050] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0051] Figure 1 The structural diagram of a data processing system provided by the present application includes: a monitored device, a front-end device and a database. The monitored device may be one or more, Figure 1 Monitored devices 1 to 3 are shown as examples, but the present application does not limit the number of monitored devices.

[0052] A monitoring module is running in the monitored device, and the monitoring module can be used to sequentially collect time series data of one or more indicators (metric) of the monitored device during operation, and send the collected time series data of the one or more indicators to the front-end device. The time series data of the indicator includes the value of the indicator and the collection time. Exemplarily, the monitoring module can collect the value of the indicator at a fixed collection interval, or collect the value of the indicator non-periodically during operation.

[0053] Exemplarily, the data processing system is applied in a cloud computing scenario, and one or more instances are running in the monitored device, and the instance may include at least one of a physical host, a virtual machine, and a container. Furthermore, a monitoring module is running in each instance, and the monitoring module can be used to sequentially collect time series data of one or more indicators of the instance in which it is located during operation, and send the collected time series data of the one or more indicators to the front-end device. Among them, the indicators can be: central processing unit (CPU) occupancy, memory occupancy, disk occupancy, disk input / output (I / O), etc.

[0054] As another example, the data processing system is applied to an IOT scenario, and the monitored device may be an edge device, such as a road camera, a road side unit (RSU), a smart street light, etc. in a smart city; or a sensor, a robot, etc. in a smart factory. The edge device may include a monitoring module, which may be used to sequentially collect time series data of one or more indicators of the edge device in which it is located during operation, and send the collected time series data of the one or more indicators to the front-end device. Among them, the indicators may be: battery saturation, position (such as altitude), temperature (such as operating temperature, ambient temperature), moving speed, etc.

[0055] It should be pointed out that the monitoring module may further include a collection thread and a time thread, the collection thread is used to collect the value of the indicator, and the time thread is used to mark the collection time (also known as marking time, marking timestamp, etc.) for the value of the indicator, or it is considered that the time thread is used to determine the time series data according to the value of the indicator and the collection time marked for the value of the indicator. It can be understood that the collection time is the collection time corresponding to the value determined by the time thread based on its own logic, and the collection time (also known as event time) when the collection thread actually collects the value of the indicator can be the same or different from the collection time marked by the time thread. The monitoring module collects the value of the indicator according to a fixed collection interval, which specifically means that the collection time for the time thread to mark the time series data has a fixed interval; the monitoring module collects the value of the indicator non-periodically, which specifically means that the collection time for the time thread to mark the time series data does not have a fixed interval.

[0056] The front-end device can obtain the time series data of one or more indicators of the monitored device during operation from the monitored device, and store the obtained time series data of each indicator in the database in sequence according to the collection time of the time series data. The front-end device can be any physical device with computing power, such as a server or a computer.

[0057] like Figure 2It is a hardware structure diagram of a front-end device, which includes a processor, a memory, a front-end interface and a back-end interface. The processor, the memory, the front-end interface and the back-end interface are all connected to a bus, and the bus can specifically be a high-speed serial computer expansion bus standard (peripheral component interconnect express, PCIe) bus. The processor is a central processing unit (CPU), which is used to obtain the time series data of one or more indicators of the monitored device during operation from the monitored device through the front-end interface, and then store the time series data of each indicator in the memory, sort the time series data of each indicator according to the acquisition time of the time series data of each indicator, and then store the sorting results in the database through the back-end interface, that is, the sorting results are persistently stored. Among them, the memory refers to the internal memory that directly exchanges data with the CPU, which can read and write data at any time, and the speed is very fast, as a temporary data storage device for the operating system or other running programs. The memory includes at least two types of memories, for example, the memory can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM). The read-only memory is a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. In addition, the memory can also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of DRAM, or a solid state disk (SSD). The memory stores a software program, and the processor runs the software program in the memory to sort the time series data of each indicator according to the collection time of the time series data of each indicator.

[0058] Optionally, the database may also be connected to an analysis device, which may be the same physical device as the front-end device, or a different physical device from the front-end device. The analysis device may read the time series data of a certain indicator over a period of time from the database, analyze the time series data of the indicator to obtain the operating status of the monitored device, and display the operating status to the tenants using the monitored device.

[0059] exist Figure 1In the data processing system shown, the protocol used to transmit time series data between the front-end device and the monitored device is usually the user datagram protocol (UDP), which cannot guarantee the sequential arrival of time series data. To this end, a time window is set in the front-end device, and the front-end device caches the time series data received within the same time window in the memory. At the end of the time window, the multiple time series data within the time window are sorted according to the acquisition time of the time series data, and the sorting results are stored in the database.

[0060] For example, the front-end device sets the time window length to 1 minute, time window 1 is from the 0th minute to the 1st minute, time window 2 is from the 1st minute to the 2nd minute, time window 3 is from the 2nd minute to the 3rd minute, etc. Further, for example, currently in time window 2, the front-end device obtains time series data (1) to time series data (5) from the monitored device, wherein the time when time series data (1) to time series data (5) arrive at the front-end device is 1.1 minutes, 1.3 minutes, 1.5 minutes, 1.7 minutes, and 1.9 minutes respectively, and the acquisition time of time series data (1) to time series data (5) is 0.3 minutes, 0.5 minutes, 0.1 minutes, 0.7 minutes, and 0.9 minutes respectively. in, then after the front-end device receives the time series data (1) to the time series data (5) in sequence, it sorts the time series data (1) to the time series data (5) according to the acquisition time of the time series data (1) to the time series data (5), and the sorting result is time series data (3), time series data (1), time series data (2), time series data (4), time series data (5), and then stores the time series data (3), time series data (1), time series data (2), time series data (4), time series data (5) in sequence into the database.

[0061] However, in this case, once the acquisition time of the time series data (6) newly acquired by the front-end device is before 0.1 minutes, the front-end device needs to read out the time series data (1) to (5) that have been stored in the database, and then reorder the time series data (6) and the time series data (1) to (5), which has a significant impact on the performance of the database.

[0062] To this end, the present application provides a data processing method and device for reducing the performance impact on the database while ensuring that the time series data is stored in the database according to the collection time of the time series data.

[0063] like Figure 3A flow chart of a data processing method exemplarily provided for the present application, the data processing method may be specifically executed by a front-end device or a module in the front-end device, and the following takes the execution of the front-end device as an example.

[0064] For the convenience of description, it is taken as an example that one of the multiple monitored devices (recorded as the target device) collects multiple time series data of an indicator. Unless otherwise specified in this application, the time series data acquired by the front-end device are all for the same indicator. Each of the multiple time series data includes the value of the indicator and the collection time corresponding to the value of the indicator. The following mainly involves the collection time corresponding to the value of the indicator included in the time series data. Furthermore, the collection time corresponding to the value of the indicator is equivalent to the collection time of the time series data.

[0065] Step 301: The front-end device obtains first time series data.

[0066] In a first possible manner, the front-end device may obtain the first time series data from the target device.

[0067] Alternatively, the front-end device may obtain the first time series data from a message middleware, wherein the message middleware is located between the front-end device and the target device, and the message middleware is, for example, Kafka. Exemplarily, the target device may send the collected first time series data to the message middleware, and the message middleware caches the first time series data. Subsequently, the front-end device reads the first time series data from the cache of the message middleware.

[0068] In the second possible way, the front-end device includes a sorting cache, and the sorting cache includes one or more time series data. Exemplarily, the multiple time series data in the sorting cache are sorted in order from early to late according to the collection time. Exemplarily, after the front-end device obtains any time series data from the target device or the message middleware, it can determine the location of the newly obtained time series data in the sorting cache according to the collection time in the time series data and the collection time in each time series data already in the sorting cache, and then store the newly obtained time series data in the sorting cache according to the location. It can be understood that the earlier the collection time of the time series data, the earlier it is in the sorting cache; and the later the collection time of the time series data, the later it is in the sorting cache. For example, the sorting cache already has time series data a to time series data c, wherein the collection time of time series data a to time series data c are 0.3 minutes, 0.5 minutes, and 0.7 minutes respectively. Accordingly, the sorting cache includes time series data a to time series data c in sequence. Furthermore, the front-end device obtains time series data d from the target device or the message middleware, and the collection time of time series data d is 0.4 minutes. Then the front-end device can determine that the location where time series data d is stored in the sorting cache is between time series data a and time series data b according to the collection time in time series data d and the collection time of time series data a to time series data c, and then store time series data d between time series data a and time series data b in the sorting cache.

[0069] Exemplarily, the sorting cache may be a memory space reserved in the memory of the front-end device for caching the time series data.

[0070] Furthermore, the front-end device may obtain the first time series data from the head of a sorting buffer (or referred to as a sorting queue) of the front-end device.

[0071] Step 302: The front-end device determines whether the first time series data is sequential data.

[0072] Specifically, when the front-end device determines that the first time series data is sequential data, the following step 303 is executed; when the front-end device determines that the first time series data is out-of-order data (ie, not sequential data), the following step 304 is executed.

[0073] In one possible manner, the front-end device determines whether the first time series data is sequential data, and specifically, the front-end device determines whether the first time series data is the next time series data after the second time series data collected by the target device. Further, when the front-end device determines that the first time series data is the next time series data after the second time series data collected by the target device, it determines that the first time series data is sequential data; when the front-end device determines that the first time series data is not the next time series data after the second time series data, it determines that the first time series data is out-of-sequence data.

[0074] The second time series data is the time series data that is most recently stored in the database by the front-end device. Exemplarily, the second time series data may be the time series data of the indicator of the target device that is first acquired by the front-end device.

[0075] In one possible manner, when the front-end device determines whether the first time series data is the next time series data after the second time series data collected by the target device, the front-end device may specifically determine whether the first time series data is the next time series data of the second time series data collected by the target device according to the first collection time, the reference collection time, and the collection rule. The reference collection time is the second collection time in the second time series data, and optionally, the reference collection time is stored in the sorting cache.

[0076] It can be understood that after the front-end device stores the second time series data in the database, it can use the second acquisition time in the second time series data as the reference acquisition time. Subsequently, the front-end device obtains the first time series data, and determines whether the first time series data is the next time series data of the second time series data according to the first acquisition time, the reference acquisition time and the acquisition rule in the first time series data.

[0077] In one possible manner, the collection rule is a rule of collecting time series data of the target device that is pre-learned by the front-end device based on multiple historical time series data. The collection time of the multiple historical time series data is all before the second collection time. The way in which the front-end device determines the collection rule can be found in Figure 6 Description in the relevant embodiments. Optionally, the front-end device may also store the collection rules in a sorting cache.

[0078] It is noted in advance that the target device can collect time series data periodically or non-periodically. For example, the target device collects time series data at 00:00:00, 00:02:00, 00:04:00 and other time points, that is, the target device collects time series data periodically, and the fixed collection interval is 2 minutes. For another example, the target device collects time series data at 00:00:02, 00:01:01, 00:01:59 and other time points, that is, the target device collects time series data non-periodically. Whether the target device collects time series data periodically is not notified to the front-end device in advance, but is learned by the front-end device based on multiple historical time series data from the target device (also see Figure 6 Description in the related embodiments).

[0079] (1) When the target device collects time series data periodically:

[0080] The collection rule includes a fixed collection interval.

[0081] When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is equal to the fixed acquisition interval, the first time series data is determined to be the next time series data of the second time series data acquired by the target device. When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is not equal to the fixed acquisition interval, the first time series data is determined not to be the next time series data of the second time series data acquired by the target device.

[0082] (2) When the target device collects time series data non-periodically:

[0083] It can be considered that the interval length of the collection time of multiple time series data conforms to the exponential distribution.

[0084] In one example, the acquisition rule includes a threshold value, and when the front-end device determines that the difference between the first acquisition time and the reference acquisition time is less than or equal to the threshold value, the first time series data is determined to be the next time series data of the second time series data acquired by the target device; when the front-end device determines that the difference between the first acquisition time and the reference acquisition time is greater than the threshold value, the first time series data is determined not to be the next time series data of the second time series data acquired by the target device. The threshold value is determined based on the average value of the exponential distribution. Exemplarily, the threshold value is determined based on the average value of the exponential distribution and the dynamic frequency tolerance, for example, the threshold value is equal to the product of the average value of the exponential distribution and the dynamic frequency tolerance.

[0085] In another example, the acquisition rule includes the average value and dynamic frequency tolerance of the exponential distribution. The front-end device can determine the threshold value based on the average value and dynamic frequency tolerance of the exponential distribution, and then when it is determined that the difference between the first acquisition time and the benchmark acquisition time is less than or equal to the threshold value, determine that the first time series data is the next time series data of the second time series data collected by the target device; and when it is determined that the difference between the first acquisition time and the benchmark acquisition time is greater than the threshold value, determine that the first time series data is the next time series data of the second time series data collected by the target device.

[0086] The dynamic frequency tolerance may be a fixed value set by the front-end device, such as the dynamic frequency tolerance being equal to 1.2.

[0087] Step 303: The front-end device stores the first time series data in a database.

[0088] Specifically, the front-end device stores the first time series data after the second time series data in the database.

[0089] In one possible embodiment, when the front-end device determines, based on the first acquisition time, the benchmark acquisition time and the acquisition rule, that the first time series data is the next time series data of the second time series data acquired by the target device, the front-end device may update the benchmark acquisition time to the first acquisition time after storing the first time series data in a database.

[0090] In one possible manner, the first time series data is the time series data obtained by the front-end device from the sorting cache, that is, the first time series data is already stored in the current sorting cache. In this case, the front-end device can also delete the first time series data in the sorting cache.

[0091] In one possible manner, the front-end device may also obtain the time series data with the earliest collection time in the sorting cache (recorded as the third time series data) from the sorting cache, wherein the third collection time of the third time series data is earlier than the collection time of other time series data in the sorting cache. If the front-end device determines that the third time series data is the next time series data of the first time series data collected by the target device, the third time series data is stored in the database and the third time series data in the sorting cache is deleted. Optionally, the front-end device also updates the reference collection time to the third collection time. The front-end device again obtains the time series data with the earliest collection time in the current sorting cache (recorded as the fourth time series data) and determines whether the fourth time series data is the next time series data of the third time series data collected by the target device. If the front-end device determines that the third time series data is not the next time series data of the first time series data collected by the target device, the third time series data continues to be stored in the sorting cache.

[0092] It should be noted that the processing method of the third time series data by the front-end device can refer to the description in steps 301 to 304, and the "first time series data" can be replaced with the "third time series data", and the "second time series data" can be replaced with the "first time series data", etc. for understanding. Similarly, the processing method of the fourth time series data by the front-end device can refer to the description in steps 301 to 304, and the "first time series data" can be replaced with the "fourth time series data", and the "second time series data" can be replaced with the "third time series data", etc. for understanding.

[0093] Step 304: The front-end device stores the first time series data into a sorting buffer of the front-end device.

[0094] Based on whether the front-end device directly obtains the first time series data from the target device or the message middleware, or obtains the first time series data from the sorting cache, the following situations are explained:

[0095] (1) When the front-end device obtains the first time series data from the target device or the message middleware:

[0096] The front-end device determines the location where the first time series data is stored in the sorting cache according to the collection time in the first time series data and the collection time of each time series data already in the sorting cache, and then stores the first time series data in the sorting cache according to the location.

[0097] After the front-end device stores the first time series data in the sorting cache, it also obtains new time series data (recorded as the fifth time series data) from the target device or the message middleware. Among them, the fifth time series data includes the fifth acquisition time. When the front-end device determines that the fifth time series data is the next time series data of the second time series data collected by the target device, the fifth time series data is stored in the database. Optionally, the front-end device also updates the benchmark acquisition time to the fifth acquisition time. When the front-end device determines that the fifth time series data is not the next time series data of the second time series data collected by the target device, the fifth time series data is stored in the sorting cache. It should be pointed out that the way the front-end device processes the fifth time series data can be referred to the description in steps 301 to 304, and the "first time series data" can be replaced with "fifth time series data" for understanding.

[0098] Figure 3 The middle process can also be considered as follows: if the front-end device determines that the new time series data is sequential data (for example, the next time series data of the time series data that was most recently stored in the database), then the front-end device stores the new time series data in the database; if it determines that the new time series data is out-of-sequence data (that is, not sequential data (for example, not the next time series data of the time series data that was most recently stored in the database)), then the front-end device stores the new time series data in the sorting cache. In the case where the front-end device stores the new time series data in the database, the front-end device also obtains the time series data with the earliest collection time from the sorting cache, and determines whether the time series data with the earliest collection time in the sorting cache is sequential data. If so, the front-end device stores the time series data with the earliest collection time in the sorting cache in the database, and deletes the time series data in the sorting cache that has been stored in the database. The front-end device then obtains the time series data with the earliest collection time from the sorting cache, and determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and so on, until there is no more time series data in the sorting cache. When the front-end device determines that the earliest time series data in the sorting cache is out-of-order data, the judgment ends. Figure 4 A flowchart of a second data processing method is shown as an example.

[0099] Combination Figure 3 and Figure 4 The following is an example of the process:

[0100] The target device periodically collects time series data to obtain time series data A to time series data E, wherein the collection time of time series data A to time series data E is 0.1min, 0.3min, 0.5min, 0.7min, and 0.9min respectively. The fixed collection period is 0.2min.

[0101] The front-end device first obtains time series data A from the target device, determines that time series data A is the first time to be obtained, and then directly stores time series data A in the database, and updates the benchmark acquisition time in the sorting cache to the acquisition time of time series data A (0.1 minute).

[0102] The front-end device continues to obtain time series data B from the target device, and determines that the acquisition time of time series data B (0.3 min) and the benchmark acquisition time (0.1 min) differ by 0.2 min, which is a fixed acquisition cycle. Therefore, it is determined that time series data B is the next time series data of time series data A collected by the target device, and the time series data B is stored after time series data A in the database.

[0103] The front-end device continues to obtain time series data D from the target device, and determines that the acquisition time of time series data D (0.7 min) differs from the benchmark acquisition time (0.3 min) by 0.4 min, that is, it is not a fixed acquisition cycle. Therefore, it is determined that time series data D is not the next time series data of time series data B collected by the target device, and the time series data D is stored in the sorting cache.

[0104] The front-end device continues to obtain time series data E from the target device, and determines that the acquisition time of time series data E (0.9 min) differs from the benchmark acquisition time (0.3 min) by 0.6 min, that is, it is not a fixed acquisition cycle. Therefore, it is determined that time series data E is not the next time series data of time series data B collected by the target device, and the time series data E is stored behind the time series data D in the sorting cache.

[0105] The front-end device continues to obtain time series data C from the target device, and determines that the acquisition time of time series data C (0.5 min) and the benchmark acquisition time (0.3 min) differ by 0.2 min, which is a fixed acquisition cycle. It is determined that time series data C is the next time series data of time series data B collected by the target device, and the time series data C is stored after the time series data B in the database, and the benchmark acquisition time is updated to 0.5 min.

[0106] Subsequently, the front-end device obtains the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data D, and determines that the collection time of the time series data D (0.7 min) and the benchmark collection time (0.5 min) differ by 0.2 min, that is, a fixed collection cycle. Therefore, it is determined that the time series data D is the next time series data of the time series data C collected by the target device, and the time series data D is stored after the time series data C in the database, the benchmark collection time is updated to 0.7 min, and the time series data D in the sorting cache is deleted.

[0107] The front-end device can also obtain the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data E, and determine that the collection time of the time series data E (0.9 min) and the benchmark collection time (0.7 min) differ by 0.2 min, which is a fixed collection cycle. Therefore, it is determined that the time series data E is the next time series data of the time series data D collected by the target device, and the time series data E is stored after the time series data D in the database, the benchmark collection time is updated to 0.9 min, and the time series data E in the sorting cache is deleted.

[0108] (2) When the front-end device obtains the first time series data from the sorting buffer:

[0109] The first time series data is the time series data at the front of the sorting cache. When the front-end device does not obtain new time series data from the target device or the message middleware, the front-end device may no longer obtain the first time series data, and determine whether the first time series data is the next time series data of the second time series data collected by the target device, so as to avoid the front-end device from performing unnecessary operations.

[0110] The front-end device obtains new time series data (i.e., the fifth time series data) from the target device or the message middleware, and stores the fifth time series data before the first time series data in the sorting buffer according to the fifth collection time of the fifth time series data. Subsequently, the front-end device obtains the fifth time series data from the sorting buffer and determines whether the fifth time series data is the next time series data of the second time series data collected by the target device.

[0111] When the above-mentioned front-end device stores the fifth time series data in the database, the front-end device can further obtain the time series data with the earliest collection time in the sorting cache (that is, the first time series data) from the sorting cache, and determine whether the first time series data is the next time series data of the fifth time series data collected by the target device.

[0112] Figure 3The middle process can also be considered as follows: if the front-end device determines that the new time series data is located before the first time series data in the sorting cache (i.e., the collection time is earlier than the earliest collection time of the time series data currently in the sorting cache), it determines whether the new time series data is sequential data (such as the next time series data of the time series data most recently stored in the database). If so, the new time series data is stored in the database. In the case where the front-end device stores the new time series data in the database, the front-end device also obtains the time series data with the earliest collection time from the sorting cache, determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and if so, the front-end device stores the time series data with the earliest collection time in the sorting cache in the database, and deletes the time series data in the sorting cache that has been stored in the database. The front-end device then obtains the time series data with the earliest collection time from the sorting cache, determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and so on, until there is no more time series data in the sorting cache. If the front-end device determines that the earliest time series data collected in the sorting cache is out-of-order data (that is, not sequential data (for example, not the next time series data of the most recent time series data stored in the database)), the judgment ends. For details, see Figure 5 A flowchart of a third data processing method is shown as an example.

[0113] Combination Figure 3 and Figure 5 In the process, the following example is provided:

[0114] The target device periodically collects time series data to obtain time series data A to time series data E, wherein the collection time of time series data A to time series data E is 0.1min, 0.3min, 0.5min, 0.7min, and 0.9min respectively. The fixed collection period is 0.2min.

[0115] The front-end device first obtains time series data A from the target device and stores time series data A in the sorting cache. Subsequently, the front-end device obtains time series data A from the sorting cache, determines that time series data A is the first time to be obtained, directly stores time series data A in the database, and updates the benchmark acquisition time in the sorting cache to the acquisition time of time series data A (0.1 minute).

[0116] The front-end device continues to obtain time series data B from the target device and stores time series data B in the sorting cache. Subsequently, the front-end device obtains time series data B from the sorting cache and determines that the acquisition time of time series data B (0.3 min) differs from the reference acquisition time (0.1 min) by 0.2 min, which is a fixed acquisition cycle. Therefore, it is determined that time series data B is the next time series data of time series data A collected by the target device, and time series data B is stored after time series data A in the database.

[0117] The front-end device continues to obtain time series data D from the target device and stores the time series data D in the sorting cache. The front-end device obtains time series data D from the sorting cache and determines that the acquisition time of time series data D (0.7 min) differs from the reference acquisition time (0.3 min) by 0.4 min, that is, it is not a fixed acquisition cycle, and determines that time series data D is not the next time series data of time series data B collected by the target device.

[0118] The front-end device continues to obtain time series data E from the target device, and according to the collection time of time series data E (0.9 minutes) and the collection time of time series data D (0.7 minutes), stores time series data E behind time series data D in the sorting cache.

[0119] The front-end device continues to obtain time series data C from the target device, and according to the acquisition time of time series data C (0.5 min), the acquisition time of time series data E (0.9 min), and the acquisition time of time series data D (0.7 min), the time series data C is stored in front of the data D in the sorting cache. Further, the front-end device determines that the acquisition time of time series data C (0.5 min) and the benchmark acquisition time (0.3 min) differ by 0.2 min, that is, a fixed acquisition cycle, and thus determines that time series data C is the next time series data of time series data B collected by the target device, stores time series data C behind time series data A in the database, deletes time series data C in the sorting cache, and updates the benchmark acquisition time to 0.5 min.

[0120] Subsequently, the front-end device obtains the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data D, and determines that the collection time of the time series data D (0.7 min) and the benchmark collection time (0.5 min) differ by 0.2 min, which is a fixed collection cycle. Therefore, it is determined that the time series data D is the next time series data of the time series data C collected by the target device, and the time series data D is stored after the time series data C in the database. The time series data D in the sorting cache is deleted, and the benchmark collection time is updated to 0.7 min.

[0121] The front-end device can also obtain the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data E, and determine that the collection time of the time series data E (0.9 min) and the benchmark collection time (0.7 min) differ by 0.2 min, which is a fixed collection cycle. Therefore, it is determined that the time series data E is the next time series data of the time series data D collected by the target device, and the time series data E is stored after the time series data D in the database. The time series data E in the sorting cache is deleted, and the benchmark collection time is updated to 0.9 min.

[0122] It should be added that, when the front-end device determines that the number of time series data stored in the sorting cache exceeds the preset number, the front-end device may consider that there is a situation of time series data loss, so the time series data corresponding to the earliest acquisition time in the sorting cache is stored in the database, and the time series data corresponding to the earliest acquisition time in the sorting cache is deleted. Subsequently, the front-end device determines the time series data corresponding to the new earliest acquisition time from the sorting cache, and determines whether the time series data corresponding to the new earliest acquisition time in the sorting cache is sequential data, such as determining whether the time series data corresponding to the new earliest acquisition time in the sorting cache is the next time series data of the time series data most recently stored in the database. If so, the front-end device stores the new time series data with the earliest acquisition time in the sorting cache in the database, and deletes the new time series data in the sorting cache that has been stored in the database. For specific implementation, please refer to the description in the above step 304. Optionally, after the front-end device stores the time series data corresponding to the earliest acquisition time in the sorting cache in the database, it can update the reference acquisition time to the earliest acquisition time.

[0123] Among them, the preset number can be a number preset by the front-end device, such as 2, 3, 5, etc.

[0124] Combined with the above example 1, assuming that the preset number is 2, after the front-end device stores the time series data E behind the time series data D in the sorting cache, the front-end device can determine that the time series data collected by the target device at the 0.5th minute has been lost, so the time series data D is stored behind the time series data B in the database, and the collection time of the time series data D (0.7th minute) is used as the new benchmark collection time. Subsequently, the front-end device determines that the difference between the collection time of the time series data E (0.9th minute) and the benchmark collection time (0.7th minute) is 0.2 minutes based on the new benchmark collection time (0.7th minute), that is, it is a fixed collection cycle, so it is determined that the time series data E is the next time series data of the time series data D collected by the target device, and the time series data E is stored behind the time series data D in the database, and the benchmark collection time is updated to 0.9th minute.

[0125] In this way, it is avoided that in the case of data loss, the front-end device keeps waiting for the lost data and cannot store the data in the sorting cache into the database in time, thereby further improving the efficiency of data processing.

[0126] Exemplarily, the front-end device includes a sorting module corresponding to the indicator, and the sorting module includes a sorting cache and a sorting execution module. The sorting execution module can be used to execute the above Figures 3 to 5 Steps in related embodiments: Exemplarily, the sorting execution module is used to obtain the first time series data, store the first time series data in a database, or store the first time series data in a sorting cache.

[0127] As described above, the implementation method of the front-end device in this application determining whether the first time series data is the next time series data of the second time series data collected by the target device according to the first collection time, the reference collection time and the collection rule is described in detail. The following explains how the front-end device pre-learns the collection rule according to multiple time series data. For details, please refer to Figure 6 A flowchart of a learning collection rule is shown as an example.

[0128] Step 601: The front-end device obtains multiple time series data. The time series data here can be considered as Figure 3 Before the related embodiments, the front-end device obtains the time series data from the target device or the message middleware, that is, the time series data here is relative to Figure 3 A related embodiment is historical time series data.

[0129] In one possible manner, the front-end device obtains multiple time series data from the target device or the message middleware, and the multiple time series data are used by the front-end device to learn the acquisition rules. After the front-end device learns the acquisition rules based on the multiple time series data, the multiple time series data can be discarded. Exemplarily, each time the front-end device obtains the time series data from the target device or the message middleware, the time series data can be cached in the training cache, that is, the front-end device caches multiple time series data in the training cache.

[0130] In another possible way, after the front-end device obtains multiple time series data from the target device or the message middleware, it not only needs to learn the collection rules based on these time series data, but also stores these time series data in the database. In this case, the front-end device can also maintain a benchmark collection time. When the front-end device obtains the collection time of new time series data, it can determine whether the collection time of the new time series data is later than the benchmark collection time. If so, the newly obtained time series data is stored in the database and the benchmark collection time is updated. Otherwise, the newly obtained time series data is directly discarded. Exemplarily, each time the front-end device obtains time series data from the target device or the message middleware, it copies the time series data and caches the copied time series data in the training cache. That is, the front-end device caches multiple time series data in the training cache. Optionally, the number of time series data cached in the training cache can be more than the number of time series data stored in the database.

[0131] Exemplarily, the training cache may be a memory space reserved in the memory of the front-end device for caching the time series data.

[0132] When the multiple time series data acquired by the front-end device are disordered data, the front-end device can first sort the multiple time series data according to the acquisition time of the multiple time series data and in the order of the acquisition time from early to late, to obtain the sorted multiple time series data, and then execute the following step 602 according to the sorted multiple time series data.

[0133] Step 602: The front-end device learns the collection time of multiple time series data to obtain a collection rule.

[0134] In one possible approach, the front-end device determines the collection interval between every two adjacent time series data (i.e., the difference between the collection times of every two adjacent time series data) based on the collection times of multiple time series data, and determines the collection rule based on the collection interval between every two adjacent time series data.

[0135] In the present application, the time series data may be N, where N is a positive integer, and the collection interval between the first time series data and the second time series data in the sorted N time series data may be recorded as f 1 , the collection interval between the second time series data and the third time series data is recorded as f 2 , the collection interval between the third time series data and the fourth time series data is recorded as f 3 , ..., the collection interval between the N-1th time series data and the Nth time series data is recorded as f N-1 .

[0136] See also Figure 7 The schematic diagram of the process of the front-end device judging whether the collection interval is regular is shown as an example:

[0137] The front-end device determines f 1 ,f 2 ,...,f N-1 When the following formulas 1 and 2 are met, it is determined that the target device collects multiple time series data based on a fixed collection interval. Among them, maxcount returns the value of the element with the most occurrences in the set f i and the number of occurrences n, α is a preset ratio. Further, the front-end device will f i The fixed collection interval is determined and written into the sorting cache as a collection rule.

[0138] f i ,n=maxcount({f 1 ,f 2 ,...,f N-1})...Formula 1

[0139]

[0140] The front-end device determines f 1 ,f 2 ,...,f N-1 If the result does not meet Formula 1 and Formula 2, it is determined that the target device does not collect multiple time series data based on a fixed collection interval. Further, it can be considered that f 1 ,f 2 ,...,f N-1 It conforms to the exponential distribution. The formula of exponential distribution is as follows: Formula 3.

[0141]

[0142] in, β is the mean of the exponential distribution. p(f;β) is the probability density function of the time interval f.

[0143] The front-end device can verify whether β is credible. When the front-end device determines that β is credible, it can be determined that the collection interval of the target device when collecting multiple time series data is consistent with the exponential distribution. For example, when the front-end device determines , it is determined that β is credible. In one example, the front-end device determines a threshold value based on β and a preset dynamic frequency tolerance δ, for example, the threshold value is equal to β×δ, and then the front-end device determines to write the threshold value into the sorting cache as a collection rule. In another example, the front-end device writes β and δ into the sorting cache as collection rules.

[0144] When the front-end device determines that β is untrustworthy, such as when the front-end device determines that It can be determined that the collection interval when the target device collects multiple time series data does not conform to the exponential distribution, that is, it is determined that the collection interval when the target device collects multiple time series data is irregular, and "irregular" is written into the sorting cache. Furthermore, the front-end device can process multiple time series data based on a water maker mechanism. Specifically, the front-end device sets a time window and an allowed delay duration. The front-end device not only sorts the multiple time series data whose arrival time is within the time window, but also sorts the time series data whose difference between the collection time and the allowed delay duration is less than or equal to the window end time as the time series data within the time window.

[0145] Exemplarily, the front-end device includes a training module corresponding to the indicator, and the training module includes a training cache and a regularity learning module. Exemplarily, the training cache has two states: a triggered state and a non-triggered state. It can be understood that when the training cache is in a non-triggered state, the regularity learning module does not need to store the time series data obtained by the front-end device from the target device or the message middleware in the training cache; when the training cache is in a triggered state, the regularity learning module needs to store the time series data obtained by the front-end device from the target device or the message middleware in the training cache. Further, the triggered state may include a triggered unfull state and a triggered full state, wherein the boundary between the triggered unfull state and the triggered full state may be whether the number of time series data included in the training cache exceeds a preset number, that is, when the number of time series data included in the training cache does not exceed the preset number, the state of the training cache is a triggered unfull state; when the number of time series data included in the training cache exceeds the preset number, the state of the training cache is a triggered full state. When determining that the training cache is in a triggered full state, the rule learning module can obtain multiple time series data from the training cache, learn according to the acquisition time of the multiple time series data, and obtain the acquisition rule. For example, the rule learning module can periodically trigger the training cache to be in a triggered state, and after obtaining the acquisition rule based on the multiple time series data of the current period, instruct the training cache to return to the non-triggered state. For details, see Figure 8 A schematic diagram showing an exemplary state switching of a training cache.

[0146] Further, when the training cache is in an untriggered state, the rule learning module determines to transparently transmit the time series data; when the training cache is in a triggered but not full state, the rule learning module copies the time series data and stores it in the training cache; further, when the training cache is in a triggered and full state, the rule learning module obtains multiple time series data from the training cache, learns according to the collection time of the multiple time series data, and obtains the collection rule. For details, please refer to Fig. 9 A schematic diagram showing another exemplary state switching of a training cache.

[0147] It should be added that the above Figures 3 to 9In the relevant embodiments, one indicator is used as an example, that is, based on multiple historical time series data of one indicator, the acquisition rule corresponding to the indicator is learned, and then the multiple time series data of the indicator are sorted according to the acquisition rule corresponding to the indicator. The front-end device in this application can be used to perform the above-mentioned Figures 3 to 9 Processing of related embodiments. Further, the front-end device can distinguish the collection rules and time series data of different indicators based on the identification information of the indicators.

[0148] Exemplarily, in a cloud computing scenario, the identification information of an indicator may include one or more of the following: region identification, availability zone (AZ) identification, tenant identification, instance identification, node identification, and metric identification. Among them, a region includes one or more availability zones, an availability zone includes one or more tenants, a tenant may deploy one or more instances, an instance may run on one or more nodes, and a node may correspond to one or more indicators. For example, the identification information of an indicator includes Beijing (region identification)--Xicheng District (availability zone identification)--Zhang San (tenant identification)--instance 1 (instance identification)--server 1 (node ​​identification)--CPU occupancy rate (metric identification), which means that the CPU occupancy rate of server 1 located in Xicheng District, Beijing, applied by Zhang San for running instance 1, needs to be collected.

[0149] In a possible implementation, the number of indicators of the target device may be K, that is, the front-end device includes K training modules (recorded as training module 1 to training module K) corresponding to the K indicators, and includes K sorting modules (recorded as sorting module 1 to sorting module K) corresponding to the K indicators, where K is an integer greater than 1. Further, for each indicator, in the training module, the rule learning module obtains multiple historical time series data from the training cache, and learns the collection rule of the indicator based on the obtained multiple historical time series data. For details, see Fig.10 The figure shows a schematic diagram of a learning collection rule under multiple indicators, wherein the rule learning module can periodically trigger the training cache to be in a triggered state, and the period of triggering the training cache in the rule learning module in different training modules can be the same or different. In the sorting module, the sorting execution module obtains the time series data of the indicator, and stores the time series data in the database in order according to the collection rule of the indicator. For details, see Fig.11A schematic diagram of data sorting under multiple indicators is shown. Optionally, the rule learning module can write the learned collection rule of the indicator into the corresponding sorting cache, and then the sorting execution module can first obtain the collection rule of the indicator from the sorting cache when determining whether to store the time series data of the indicator into the database.

[0150] In this application, the front-end device can continuously obtain time series data from the monitored device, and the time series data obtained by the front-end device can be considered as streaming data, that is, the streaming data includes multiple time series data. Fig.12 The flowchart of the fourth data processing method provided by the present application is mainly based on the example of the front-end device obtaining streaming data. For any content not described in detail, please refer to Figures 3 to 11 The description in the relevant embodiments is as follows:

[0151] The training module 1 determines that the collection rule 1 includes a fixed collection interval of 1 minute based on the historical streaming data 1, and writes the fixed collection interval of 1 minute into the sorting cache 1 of the sorting module 1. Furthermore, the sorting cache 1 also includes a reference collection time of 10:00:00, indicating that the collection time of the time series data in the streaming data 1 last stored in the database was 10:00:00. When the sorting module 1 obtains new time series data in the streaming data 1 from the target device or the message middleware again, it determines whether the new time series data is the next time series data of the time series data last stored in the database based on the fixed collection interval of 1 minute and the reference collection time of 10:00:00.

[0152] The training module 2 determines that the collection rule 2 includes the exponential distribution mean value 1min and the dynamic frequency tolerance 1.1 based on the historical streaming data 2, and writes the exponential distribution mean value 1min and the dynamic frequency tolerance 1.1 into the sorting cache 2 of the sorting module 2. Furthermore, the sorting cache 2 also includes the benchmark collection time 10:00:10, indicating that the collection time of the time series data in the streaming data 2 last stored in the database is 10:00:10. When the sorting module 2 obtains new time series data in the streaming data 2 from the target device or the message middleware again, it determines whether the new time series data is the next time series data of the time series data last stored in the database based on the exponential distribution mean value 1min, the dynamic frequency tolerance 1.1 and the benchmark collection time 10:00:10.

[0153] The training module K determines that the collection of historical streaming data K is irregular, and writes "irregularity" into the sorting cache K of the sorting module K. Furthermore, the sorting cache K also includes a time window length of 10 minutes and an allowed delay time of 2 minutes. The sorting module K can store multiple time series data sequentially in the database based on the watermark mechanism, the time window length of 10 minutes and the allowed delay time of 2 minutes.

[0154] It should also be added that the time series data of which indicators are collected by the target device is specifically selected by the tenant. In a specific implementation, the tenant can send a collection request to the front-end device, and accordingly, the front-end device receives the collection request from the tenant, and the collection request includes the identification information of the indicator. The front-end device determines the target device from multiple monitored devices based on the identification information of the indicator, and then deploys the monitoring module of the indicator in the target device. The monitoring module is used for the target device to sequentially collect multiple time series data of the indicator during the operation of the target device. In a specific implementation, the tenant can log in to the client and send a collection request to the front-end device by operating the client.

[0155] Optionally, the tenant may also send an analysis request to the analysis device. Accordingly, the analysis device receives the analysis request from the tenant, and the analysis request includes the identification information of the indicator and the analysis period. Based on the identification information and the analysis period of the indicator, the analysis device obtains the time series data of the indicator whose collection time is within the analysis period from the database, and then analyzes the operating status of the indicator based on the obtained time series data of the indicator, and displays the obtained operating status of the target device to the tenant. In a specific implementation, the tenant can log in to the client and send the analysis request to the analysis device by operating the client. Accordingly, the analysis device sends the operating status of the target device to the client, and the client displays the operating status to the tenant on the display interface of the client.

[0156] In the above technical solution, after the front-end device determines to obtain the first time series data, it needs to determine whether the first time series data is sequential data. When it is determined that the first time series data is sequential data, the first data will be stored in the database, and when it is determined that the first time series data is out of order data, the first time series data will be stored in the sorting cache of the front-end device. In this way, the front-end device is prevented from storing out of order data in the database. When the acquisition time of the newly received data is earlier than the acquisition time of the time series data already stored in the database, the front-end device reads the time series data in the database again, and sorts the read data and the newly received data, which affects the database performance. That is, this method can reduce the impact on database performance, reduce resource consumption for data processing, and reduce the problem of write amplification.

[0157] In addition, when the front-end device determines that the first time series data is sequential data, it will store the first data in the database without first storing the first time series in the sorting cache, and sorting the multiple time series data in the sorting cache whose acquisition time is within the preset time window before storing them in the database, thereby eliminating the problem of the front-end device storing the time series data untimely, that is, helping to improve the efficiency of data processing.

[0158] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the protection scope of this application.

[0159] Based on the above content and the same concept, Fig.13 A data processing device is exemplarily provided for the present application, and the device is applied to a front-end device of a data processing system. Exemplarily, the device can be a front-end device or a module in the front-end device.

[0160] The device includes: an acquisition module 1301, a judgment module 1302 and a storage module 1303. Among them, the acquisition module 1301 is used to acquire the first time series data; the judgment module 1302 is used to determine whether the first time series data is sequential data; the storage module 1303 is used to store the first time series data in a database when the judgment module 1302 determines that the first time series data is sequential data; or, when the judgment module 1302 determines that the first time series data is out of order data, the first time series data is stored in the sorting cache of the front-end device. In this application, the acquisition module 1301, the judgment module 1302, and the storage module 1303 can be collectively referred to as a sorting execution module.

[0161] In a possible implementation, when determining whether the first time series data is sequential data, the determination module 1302 is specifically used to determine whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

[0162] In one possible implementation, after the storage module 1303 stores the first time series data in the database, the acquisition module 1301 is further used to: obtain third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache whose collection time is earlier than other time series data; the judgment module 1302 is further used to: determine whether the third time series data is sequential data; the storage module 1303 is further used to: store the third time series data in the database when the judgment module 1302 determines that the third time series data is sequential data.

[0163] In a possible implementation, after storing the first time series data in the sorting cache of the front-end device, the storage module 1303 is further used to: when it is determined that the number of time series data in the sorting cache exceeds a preset number, store the time series data in the sorting cache that is collected earlier than other time series data in the database.

[0164] In a possible implementation, when the storage module 1303 stores the first time series data in the sorting cache of the front-end device, it is specifically used to: determine the position of storing the first time series data in the sorting cache according to the first acquisition time of the first time series data and the acquisition time of the time series data already in the sorting cache; and store the first time series data in the sorting cache according to the position.

[0165] In one possible implementation, when judging whether the first time series data is the next time series data after the second time series data, the judgment module 1302 is specifically used to: judge whether the first time series data is the next time series data after the second time series data based on a benchmark collection time, a collection rule, and a first collection time of the first time series data, wherein the collection rule is a rule for collecting time series data of a target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

[0166] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the judgment module 1302 judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, determine that the first time series data is the next time series data after the second time series data.

[0167] In one possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the average value and dynamic frequency tolerance of the exponential distribution; when the judgment module 1302 judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, determine that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value and dynamic frequency tolerance of the exponential distribution.

[0168] In a possible implementation, after storing the first time series data in the database, the determination module 1302 is further configured to: update the reference collection time to the first collection time of the first time series data.

[0169] Among them, the acquisition module 1301, the judgment module 1302 and the storage module 1303 can all be implemented by software, or can be implemented by hardware. Exemplarily, the following takes the acquisition module 1301 as an example to introduce the implementation of the acquisition module 1301. Similarly, the implementation of the judgment module 1302 and the storage module 1303 can refer to the implementation of the acquisition module 1301.

[0170] As an example of a software functional unit, the acquisition module 1301 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the acquisition module 1301 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region (region) or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, a region may generally include multiple AZs. Similarly, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Among them, usually a VPC is set in a region. The cross-region communication between two VPCs in the same region and between VPCs in different regions requires setting up a communication gateway in each VPC to achieve interconnection between VPCs through the communication gateway.

[0171] As an example of a hardware functional unit, the acquisition module 1301 may include at least one computing device, such as a server, etc. Alternatively, the acquisition module 1301 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0172] The multiple computing devices included in the acquisition module 1301 can be distributed in the same region or in different regions. The multiple computing devices included in the acquisition module 1301 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the acquisition module 1301 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0173] It should be noted that, in other embodiments, the acquisition module 1301 may be used to execute Figures 3 to 12 For any step in the related method embodiment, the judgment module 1302 can be used to execute Figures 3 to 12 For any step in the related method embodiment, the storage module 1303 can be used to execute Figures 3 to 12 Any steps in the relevant method embodiments, that is, the steps implemented by the acquisition module 1301, the judgment module 1302, and the storage module 1303 can be specified as needed, and the acquisition module 1301, the judgment module 1302, and the storage module 1303 are respectively executed. Figures 3 to 12 The different steps in the relevant method embodiments are used to implement all functions of the data processing device.

[0174] The present application also provides a computing device 100. Fig.14 As shown, the computing device 100 includes: a processor 101, a memory 102, a communication interface 103 and a bus 104. The processor 101, the memory 102 and the communication interface 103 communicate through the bus 104. The computing device 100 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in the computing device 100.

[0175] The bus 104 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.14 The bus 104 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 104 may include a path for transmitting information between various components of the computing device 100 (eg, the memory 102, the processor 101, and the communication interface 103).

[0176] The processor 101 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP) or a digital signal processor (DSP).

[0177] The memory 102 may include a volatile memory, such as a random access memory (RAM). The processor 101 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid state drive (SSD).

[0178] The memory 102 stores executable program codes, and the processor 101 executes the executable program codes to respectively implement the functions of the aforementioned acquisition module 1301, the judgment module 1302, and the storage module 1303, thereby implementing Figures 3 to 12 The method in the related method embodiment. That is, the memory 102 stores the method for executing Figures 3 to 12 Instructions of the method in the related method embodiments.

[0179] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0180] The embodiment of the present application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smart phone.

[0181] like Fig.15 As shown, the computing device cluster includes at least one computing device 100. The memory 102 in one or more computing devices 100 in the computing device cluster may store the same Figures 3 to 12 Instructions of the methods in the related method embodiments.

[0182] In some possible implementations, the memory 102 of one or more computing devices 100 in the computing device cluster may also store a program for executing Figures 3 to 12 In other words, a combination of one or more computing devices 100 can jointly execute Figures 3 to 12 Instructions of the methods in the related method embodiments.

[0183] It should be noted that the memory 102 in different computing devices 100 in the computing device cluster can store different instructions, which are respectively used to execute part of the functions of the data processing device. That is, the instructions stored in the memory 102 in different computing devices 100 can implement the functions of one or more modules in the acquisition module 1301, the judgment module 1302 and the storage module 1303.

[0184] In some possible implementations, one or more computing devices in the computing device cluster may be connected via a network, which may be a wide area network or a local area network. Fig.16 A possible implementation is shown. Fig.16 As shown, two computing devices 100A and 100B are connected via a network. Specifically, they are connected to the network via a communication interface in each computing device. In this type of possible implementation, the memory 102 in the computing device 100A stores instructions for executing the functions of the acquisition module 1301. At the same time, the memory 102 in the computing device 100B stores instructions for executing the functions of the judgment module 1302 and the storage module 1303.

[0185] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes Figures 3 to 12 The method in the related method embodiment.

[0186] The present application also provides a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute Figures 3 to 12 The method in the related method embodiment.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a front-end device of a data processing system, the data processing system further comprising a target device and a database, the target device being used to collect a plurality of time series data during the operation of the target device, each time series data comprising the collection time of the time series data, the method comprising: Acquire first time series data; Determining whether the first time series data is sequential data; When it is determined that the first time series data is sequential data, storing the first time series data in the database; Alternatively, when it is determined that the first time series data is out-of-order data, the first time series data is stored in a sorting cache of the front-end device.

2. The method according to claim 1, characterized in that The determining whether the first time series data is sequential data includes: It is determined whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

3. The method according to claim 1 or 2, characterized in that After storing the first time series data in the database, the method further includes: Acquire third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache that is collected earlier than other time series data; Determining whether the third time series data is sequential data; In a case where it is determined that the third time series data is sequential data, the third time series data is stored in the database.

4. The method according to any one of claims 1 to 3, characterized in that After storing the first time series data in the sorting cache of the front-end device, the method further includes: When it is determined that the number of time series data in the sorting cache exceeds a preset number, the time series data in the sorting cache whose collection time is earlier than other time series data is stored in the database.

5. The method according to any one of claims 1 to 4, characterized in that The storing the first time series data into the sorting cache of the front-end device includes: Determine a location in the sorting buffer where to store the first time series data according to a first collection time of the first time series data and a collection time of time series data already in the sorting buffer; According to the position, the first time series data is stored in the sorting cache.

6. The method according to any one of claims 2 to 5, characterized in that The determining whether the first time series data is the next time series data after the second time series data includes: Based on the benchmark collection time, the collection rule and the first collection time of the first time series data, determine whether the first time series data is the next time series data after the second time series data, wherein the collection rule is the rule for collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

7. The method according to claim 6, characterized in that The multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; The determining, according to the reference collection time, the collection rule, and the first collection time of the first time series data, whether the first time series data is the next time series data after the second time series data includes: When the difference between the first acquisition time and the reference acquisition time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data.

8. The method according to claim 6, characterized in that The collection time intervals of the plurality of historical time series data conform to an exponential distribution, and the collection rule includes an average value and a dynamic frequency tolerance of the exponential distribution; The determining, according to the reference collection time, the collection rule, and the first collection time of the first time series data, whether the first time series data is the next time series data after the second time series data includes: When it is determined that the difference between the first acquisition time and the reference acquisition time is less than or equal to a threshold, it is determined that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value of the exponential distribution and the dynamic frequency tolerance.

9. The method according to any one of claims 6 to 8, characterized in that After storing the first time series data in the database, the method further includes: The reference acquisition time is updated to be the first acquisition time of the first time series data.

10. A data processing device, characterized in that: The device is applied to a front-end device of a data processing system, the data processing system further comprising a target device and a database, the target device being used to collect a plurality of time series data during the operation of the target device, each time series data comprising the collection time of the time series data, the device comprising: An acquisition module, used for acquiring first time series data; A judging module, used for judging whether the first time series data is sequential data; A storage module is used to store the first time series data in the database when the judgment module determines that the first time series data is sequential data; or to store the first time series data in the sorting cache of the front-end device when the judgment module determines that the first time series data is out-of-order data.

11. The device according to claim 10, characterized in that When the judgment module judges whether the first time series data is sequential data, it is specifically used to: It is determined whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

12. The device according to claim 10 or 11, characterized in that After the storage module stores the first time series data in the database, The acquisition module is further used to: acquire third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache whose collection time is earlier than other time series data; The judging module is further used to: judge whether the third time series data is sequential data; The storage module is further configured to: when the judgment module determines that the third time series data is sequential data, store the third time series data in the database.

13. The device according to any one of claims 10 to 12, characterized in that After storing the first time series data in the sorting cache of the front-end device, the storage module is further used to: When it is determined that the number of time series data in the sorting cache exceeds a preset number, the time series data in the sorting cache whose collection time is earlier than other time series data is stored in the database.

14. The device according to any one of claims 10 to 13, characterized in that When the storage module stores the first time series data in the sorting cache of the front-end device, the storage module is specifically used to: Determine a location in the sorting buffer where to store the first time series data according to a first collection time of the first time series data and a collection time of time series data already in the sorting buffer; According to the position, the first time series data is stored in the sorting cache.

15. The device according to any one of claims 11 to 14, characterized in that When the judgment module judges whether the first time series data is the next time series data after the second time series data, it is specifically used to: Based on the benchmark collection time, the collection rule and the first collection time of the first time series data, determine whether the first time series data is the next time series data after the second time series data, wherein the collection rule is the rule for collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

16. The device according to claim 15, characterized in that The multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; When the judgment module judges whether the first time series data is the next time series data after the second time series data according to the reference collection time, the collection rule and the first collection time of the first time series data, the judgment module is specifically used to: When the difference between the first acquisition time and the reference acquisition time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data.

17. The device according to claim 15, characterized in that The collection time intervals of the plurality of historical time series data conform to an exponential distribution, and the collection rule includes an average value and a dynamic frequency tolerance of the exponential distribution; When the judgment module judges whether the first time series data is the next time series data after the second time series data according to the reference collection time, the collection rule and the first collection time of the first time series data, the judgment module is specifically used to: When it is determined that the difference between the first acquisition time and the reference acquisition time is less than or equal to a threshold, it is determined that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value of the exponential distribution and the dynamic frequency tolerance.

18. The device according to any one of claims 15 to 17, characterized in that After storing the first time series data in the database, the judgment module is further configured to: update the reference acquisition time to the first acquisition time of the first time series data.

19. A data processing system, characterized in that: include: Front-end devices, target devices and databases; The target device is used to collect a plurality of time series data during the operation of the target device, each time series data includes a collection time of the time series data; The front-end device is used to execute the method according to any one of claims 1 to 9.

20. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

21. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images