Data processing method and apparatus

Abstract

A data processing method and apparatus, which are applied to a front-end device in a data processing system. The data processing system further comprises a target device and a database, wherein the target device is configured to collect a plurality of pieces of time series data in an operation process of the target device, and each piece of time series data comprises a collection time for the time series data. The method comprises: a front-end device acquiring first time series data, and determining whether the first time series data is sequential data; further, when determining that the first time series data is sequential data, the front-end device storing the first time series data in a database; alternatively, when determining that the first time series data is out-of-order data, the front-end device storing the first time series data in a sort buffer of the front-end device. In this way, the impact on the performance of the database caused by the front-end device storing the out-of-order data in the database is avoided.

Description

Data processing method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on July 12, 2023, with application number 202310854392.7 and application name “A method, device and computing device cluster for processing time series data”, the entire contents of which are incorporated by reference into this application; this application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 30, 2023, with application number 202311427094.6 and application name “A data processing method and device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The embodiments of the present application relate to the field of cloud computing, and in particular to a data processing method and device. Background Art

[0004] Time series data refers to data columns recorded in chronological order. Each data point within the same data column can represent operational data for the same metric and must be comparable. Time series data is very common in the Internet of Things (IoT), cloud computing operations, and finance. After being sampled by the sampling end, time series data typically needs to be transmitted to the collection end for unified analysis and processing.

[0005] However, due to factors such as network latency, the order in which time series data arrives at the collection end is often inconsistent with the order in which it is sorted based on the time of collection. This type of time series data is called out-of-order data. After the collection end stores multiple time series data in the database, if out-of-order data arrives, this newly arrived out-of-order data cannot be directly stored in the database. Instead, the database must first retrieve the multiple historical time series data preceding the out-of-order data. These retrieved historical time series data and the out-of-order data must then be reordered before storage.

[0006] This has a significant impact on database performance.

[0007] Summary of the Invention

[0008] The present application provides a data processing method and apparatus for reducing the impact of out-of-order data on database performance.

[0009] In a first aspect, the present application provides a data processing method, which is applied to a front-end device of a data processing system. For example, the method can be executed by the front-end device or a module in the front-end device. For ease of description, the following description uses the front-end device executing the method as an example.

[0010] The data processing system also includes a target device and a database. The target device is used to collect multiple time series data during the operation of the target device. Each time series data includes the collection time of the time series data.

[0011] The method includes: a front-end device acquiring first time series data and determining whether the first time series data is sequential data. Furthermore, if the front-end device determines that the first time series data is sequential data, the front-end device stores the first time series data in a database; or, if the front-end device determines that the first time series data is out-of-order data, the front-end device stores the first time series data in a sorting cache of the front-end device.

[0012] In the above technical solution, after obtaining the first time series data, the front-end device needs to determine whether the first time series data is sequential data. When it is determined that the first time series data is sequential data, the first data will be stored in the database. When it is determined that the first time series data is out of order data, the first time series data will be stored in the sorting cache of the front-end device. In this way, the front-end device is prevented from storing out of order data in the database. When the acquisition time of the newly received data is earlier than the acquisition time of the time series data already stored in the database, the front-end device reads the time series data in the database again and sorts the read data and the newly received data, thereby reducing the impact on the database performance. That is, this method can reduce the impact on the database performance, reduce the resource consumption of data processing, and reduce the problem of write amplification.

[0013] In addition, when the front-end device determines that the first time series data is sequential data, it will store the first time series data in the database, without first storing the first time series data in the sorting cache, and sorting the multiple time series data in the sorting cache whose arrival time is within the preset time window before storing them in the database, thereby solving the problem of untimely processing of time series data. That is, it helps to improve the efficiency of data processing.

[0014] In one possible implementation, when the front-end device determines whether the first time series data is sequential data, it may specifically determine whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device. Furthermore, when the front-end device determines that the first time series data is the next time series data after the second time series data, it determines that the first time series data is sequential data; and when the front-end device determines that the first time series data is not the next time series data after the second time series data, it determines that the first time series data is out of sequence data. As described above, a method for the front-end device to determine whether the first time series data is sequential data is provided.

[0015] In one possible implementation, after storing the first time series data in the database, the front-end device can also obtain third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache that is collected earlier than other time series data. The front-end device determines whether the third time series data is sequential data, and if it is determined that the third time series data is sequential data, it stores the third time series data in the database. In the above technical solution, once the front-end device stores the first time series data in the database, it can obtain time series data (i.e., the third time series data) that is collected earlier than other time series data from the sorting cache, and then, after determining that the third time series data is sequential data, write the third time series data to the database, which helps to further improve the efficiency of data processing.

[0016] In one possible implementation, after the front-end device stores the first time series data in the sorting cache of the front-end device, if it determines that the number of time series data in the sorting cache exceeds a preset number, the front-end device stores the time series data in the sorting cache that is collected earlier than the other time series data in the database. In the above technical solution, if the front-end device determines that the number of time series data included in the sorting cache exceeds the preset number, it determines that a certain time series data that is collected earlier than all the time series data in the sorting cache has been lost, and then the time series data in the sorting cache that is collected earlier than the other time series data is stored in the database, thereby avoiding the problem of the front-end device constantly waiting for a certain lost time series data, resulting in prolonged data processing time, that is, helping to further improve the efficiency of data processing.

[0017] In one possible implementation, when a front-end device stores the first time series data in a sorting cache of the front-end device, the front-end device may specifically determine the location in the sorting cache where to store the first time series data based on the first acquisition time of the first time series data and the acquisition time of the time series data already in the sorting cache; and store the first time series data in the sorting cache based on the location. In the above technical solution, the front-end device sorts the multiple time series data in the sorting cache based on the acquisition time, which helps the front-end device quickly determine the time series data with the earliest acquisition time from the sorting cache, thereby helping to further improve data processing efficiency.

[0018] In one possible implementation, when the front-end device determines whether the first time series data is the next time series data after the second time series data, specifically, the front-end device can determine whether the first time series data is the next time series data after the second time series data based on the benchmark acquisition time, the acquisition rule and the first acquisition time of the first time series data, wherein the acquisition rule is the rule for the target device to acquire time series data that is pre-learned based on multiple historical time series data, and the benchmark acquisition time is the second acquisition time of the second time series data. In the above technical solution, the front-end device can obtain the acquisition rule, which is the pre-learned rule for the target device to acquire time series data. Furthermore, the front-end device can use the acquisition time of the previous time series data (i.e., the second time series data) stored in the database as the benchmark acquisition time, and then determine whether the first time series data is the next time series data after the second time series data based on the benchmark acquisition time, the acquisition rule and the first acquisition time of the first time series data, which helps to improve the accuracy of the judgment.

[0019] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the front-end device determines whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it can be specifically that when the front-end device determines that the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data; when the front-end device determines that the difference between the first collection time and the benchmark collection time is not equal to the fixed time interval, it is determined that the first time series data is not the next time series data after the second time series data. In one possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the mean value and dynamic frequency tolerance of the exponential distribution; when the front-end device determines whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule, and the first collection time of the first time series data, it can be specifically determined that when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, the first time series data is determined to be the next time series data of the second time series data; when it is determined that the difference between the first collection time and the benchmark collection time is greater than a threshold, the first time series data is determined not to be the next time series data of the second time series data, wherein the threshold is determined by the mean value and dynamic frequency tolerance of the exponential distribution. As described above, providing two different collection rules and an implementation method in which the front-end device determines whether the first time series data is the next time series data of the second time series data based on the collection rule helps to improve the accuracy of the judgment.

[0020] In one possible implementation, after storing the first time series data in the database, the front-end device further updates the reference acquisition time to the first acquisition time of the first time series data. In the above technical solution, the front-end device can update the reference acquisition time based on the acquisition time of the first time series data most recently stored in the database, and then determine whether the next time series data is sequential data (i.e., the time series data next to the first time series data acquired by the target device) based on the updated reference acquisition time, which helps improve the accuracy of the judgment.

[0021] In a second aspect, the present application provides a data processing apparatus, which is applied to a front-end device of a data processing system. The data processing system also includes a target device and a database. The target device is configured to collect multiple time series data during the operation of the target device, each time series data including the time when the time series data was collected. Exemplarily, the apparatus can be a front-end device or a module within the front-end device.

[0022] The device includes: an acquisition module for acquiring first time series data; a judgment module for judging whether the first time series data is sequential data; a storage module for storing the first time series data in a database when the judgment module determines that the first time series data is sequential data; or, when the judgment module determines that the first time series data is out-of-order data, storing the first time series data in a sorting cache of a front-end device.

[0023] In one possible implementation, when the judgment module judges whether the first time series data is sequential data, it is specifically used to: judge whether the first time series data is the next time series data after the second time series data, and the second time series data is the time series data most recently stored in the database by the front-end device.

[0024] In one possible implementation, after the storage module stores the first time series data in the database, the acquisition module is further used to: obtain third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache that is collected earlier than other time series data; the judgment module is further used to: determine whether the third time series data is sequential data; the storage module is further used to: store the third time series data in the database if the judgment module determines that the third time series data is sequential data.

[0025] In one possible implementation, after storing the first time series data in the sorting cache of the front-end device, the storage module is further used to: when it is determined that the number of time series data in the sorting cache exceeds a preset number, store the time series data in the sorting cache that is collected earlier than other time series data in the database.

[0026] In one possible implementation, when the storage module stores the first time series data in the sorting cache of the front-end device, it is specifically used to: determine the position for storing the first time series data in the sorting cache based on the first acquisition time of the first time series data and the acquisition time of the existing time series data in the sorting cache; and store the first time series data in the sorting cache based on the position.

[0027] In one possible implementation, when the judgment module judges whether the first time series data is the next time series data after the second time series data, it is specifically used to: judge whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, wherein the collection rule is the rule of collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

[0028] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the judgment module judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, determine that the first time series data is the next time series data after the second time series data.

[0029] In one possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the average value and dynamic frequency tolerance of the exponential distribution; when the judgment module judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, determine that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value and dynamic frequency tolerance of the exponential distribution.

[0030] In a possible implementation, after storing the first time series data in the database, the judgment module is further configured to: update the reference collection time to the first collection time of the first time series data.

[0031] In a third aspect, the present application provides a data processing system comprising: a front-end device, a target device and a database; wherein the target device is used to collect multiple time series data during the operation of the target device, and each time series data includes the collection time of the time series data; the front-end device is used to execute the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0032] In a fourth aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0033] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the computing device cluster to execute the method in the above-mentioned first aspect or any possible implementation manner of the first aspect.

[0034] In a sixth aspect, the present application provides a computer-readable storage medium comprising computer program instructions. When the computer program instructions are executed by a computing device cluster, the computing device cluster executes the method in the above-mentioned first aspect or any possible implementation of the first aspect.

[0035] The technical effects that can be achieved in any of the second to sixth aspects mentioned above can refer to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic structural diagram of a data processing system;

[0037] FIG2 is a schematic diagram of the hardware structure of a front-end device;

[0038] FIG3 is a flow chart of a first data processing method provided by the present application;

[0039] FIG4 is a flow chart of a second data processing method provided by the present application;

[0040] FIG5 is a flow chart of a third data processing method provided by this application;

[0041] FIG6 is a schematic diagram of a flow chart of a learning collection rule provided by the present application;

[0042] FIG7 is a schematic diagram of a flow chart of a front-end device provided by the present application for determining whether a collection interval is regular;

[0043] FIG8 is a schematic diagram of a state switching of a training cache provided by the present application;

[0044] FIG9 is a schematic diagram of another state switching of a training cache provided by the present application;

[0045] FIG10 is a schematic diagram of a learning and collection rule under multiple indicators provided by this application;

[0046] FIG11 is a schematic diagram of data sorting under multiple indicators provided by this application;

[0047] FIG12 is a schematic diagram of a fourth data processing method provided by this application;

[0048] FIG13 is a schematic structural diagram of a data processing device provided by the present application;

[0049] FIG14 is a schematic diagram of the structure of a computing device provided by the present application;

[0050] FIG15 is a schematic diagram of the structure of a computing device cluster provided by the present application;

[0051] FIG16 is a schematic diagram of network connections in a computing device cluster provided by this application. DETAILED DESCRIPTION

[0052] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0053] Figure 1 is a schematic diagram of the structure of a data processing system provided by this application, which includes: a monitored device, a front-end device, and a database. There can be one or more monitored devices. Figure 1 exemplarily shows monitored devices 1 through 3, but this application does not limit the number of monitored devices.

[0054] The monitored device runs a monitoring module that can sequentially collect time series data of one or more metrics of the monitored device during operation and send the collected time series data of the one or more metrics to the front-end device. The time series data of the metric includes the value of the metric and the collection time. For example, the monitoring module can collect the value of the metric at a fixed collection interval, or collect the value of the metric non-periodically during operation.

[0055] Exemplarily, the data processing system is applied in a cloud computing scenario, where one or more instances are running in the monitored device, and the instances may include at least one of a physical host, a virtual machine, and a container. Furthermore, a monitoring module is running in each instance, and the monitoring module can be used to sequentially collect time series data of one or more indicators of the instance in which it is located during operation, and send the collected time series data of the one or more indicators to the front-end device. Among them, the indicators can be: central processing unit (CPU) occupancy, memory occupancy, disk occupancy, disk input / output (I / O), etc.

[0056] As another example, the data processing system is applied to an IOT scenario, and the monitored devices may be edge devices, such as road cameras, road side units (RSUs), smart street lights, etc. in smart cities; or sensors and robots in smart factories. The edge device may include a monitoring module, which can be used to sequentially collect time series data of one or more indicators of the edge device during operation, and send the collected time series data of the one or more indicators to the front-end device. Among them, the indicators may be: battery saturation, location (such as altitude), temperature (such as operating temperature, ambient temperature), moving speed, etc.

[0057] It should be pointed out that the monitoring module may further include a collection thread and a time thread. The collection thread is used to collect the value of the indicator, and the time thread is used to mark the collection time (also known as marking time, marking timestamp, etc.) for the value of the indicator, or it can be considered that the time thread is used to determine the time series data based on the value of the indicator and the collection time it marks the value of the indicator. It can be understood that the collection time is the collection time corresponding to the value determined by the time thread based on its own logic, and the collection time (also known as event time) when the collection thread actually collects the value of the indicator can be the same as or different from the collection time marked by the time thread. The monitoring module collects the value of the indicator according to a fixed collection interval, specifically, the collection time for the time thread to mark the time series data has a fixed interval; the monitoring module collects the value of the indicator non-periodically, specifically, the collection time for the time thread to mark the time series data does not have a fixed interval.

[0058] The front-end device can obtain time series data for one or more indicators of the monitored device during its operation, and store the obtained time series data for each indicator in the database in order of the time series data collection time. The front-end device can be any physical device with computing power, such as a server or computer.

[0059] Figure 2 is a schematic diagram of the hardware structure of a front-end device. The front-end device includes a processor, memory, front-end interface, and back-end interface. The processor, memory, front-end interface, and back-end interface are all connected to a bus, which can specifically be a high-speed serial computer expansion bus standard (Peripheral Component Interconnect Express, PCIe) bus. The processor is a central processing unit (CPU) that is used to obtain time series data of one or more indicators of the monitored device during operation from the monitored device through the front-end interface, then store the time series data of each indicator in the memory, sort the time series data of each indicator according to the acquisition time of the time series data of each indicator, and then store the sorting results in a database through the back-end interface, that is, to store the sorting results persistently. Among them, the memory refers to the internal memory that directly exchanges data with the CPU. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. The memory includes at least two types of memory. For example, the memory can be either a random access memory or a read-only memory (ROM). For example, random access memory is dynamic random access memory (DRAM) or storage class memory (SCM). Read-only memory is programmable read only memory (PROM), erasable programmable read only memory (EPROM), etc. In addition, the memory can also be a dual in-line memory module or a dual in-line memory module (DIMM), that is, a module composed of DRAM, or a solid state disk (SSD). The memory stores a software program, and the processor runs the software program in the memory to sort the time series data of each indicator according to the collection time of the time series data of each indicator.

[0060] Optionally, the database may also be connected to an analysis device, which may be the same physical device as the front-end device, or a different physical device from the front-end device. The analysis device may read the time series data of a certain indicator over a period of time from the database, analyze the time series data of the indicator to obtain the operating status of the monitored device, and display the operating status to the tenants using the monitored device.

[0061] In the data processing system shown in Figure 1, the protocol used to transmit time series data between the front-end device and the monitored device is usually the User Datagram Protocol (UDP). UDP cannot guarantee the sequential arrival of time series data. To this end, a time window is set in the front-end device. The front-end device caches the time series data received within the same time window in memory. At the end of the time window, the multiple time series data within the time window are sorted according to the acquisition time of the time series data, and the sorting results are stored in the database.

[0062] For example, the front-end device sets the time window length to 1 minute, time window 1 is from the 0th minute to the 1st minute, time window 2 is from the 1st minute to the 2nd minute, time window 3 is from the 2nd minute to the 3rd minute, etc. Further, for example, currently in time window 2, the front-end device obtains time series data (1) to time series data (5) from the monitored device, wherein the time when time series data (1) to time series data (5) arrive at the front-end device is 1.1 minutes, 1.3 minutes, 1.5 minutes, 1.7 minutes, and 1.9 minutes respectively, while the acquisition time of time series data (1) to time series data (5) is 0.3 minutes, 0.5 minutes, 0.1 minutes, 0.7 minutes, and 0.9 minutes respectively. in, then after the front-end device receives the time series data (1) to the time series data (5) in sequence, it sorts the time series data (1) to the time series data (5) according to the acquisition time of the time series data (1) to the time series data (5), and the sorting result is time series data (3), time series data (1), time series data (2), time series data (4), time series data (5), and then stores the time series data (3), time series data (1), time series data (2), time series data (4), time series data (5) in the database in sequence.

[0063] However, in this case, once the acquisition time of the newly acquired time series data (6) by the front-end device is before 0.1 minutes, the front-end device needs to read out the time series data (1) to time series data (5) that have been stored in the database, and then re-sort the time series data (6) and time series data (1) to time series data (5). This has a greater impact on the performance of the database.

[0064] To this end, the present application provides a data processing method and device for reducing the performance impact on the database while ensuring that time series data is stored in the database according to the collection time of the time series data.

[0065] As shown in Figure 3, a flow chart of a data processing method exemplarily provided in this application, the data processing method can be specifically executed by a front-end device or a module in the front-end device. The following takes the execution of the front-end device as an example.

[0066] For the convenience of description, an example is given of a monitored device (denoted as the target device) among multiple monitored devices collecting multiple time series data of an indicator. Unless otherwise specified in this application, the time series data obtained by the front-end device are all for the same indicator. Each time series data in the multiple time series data includes the value of the indicator and the collection time corresponding to the value of the indicator. The following mainly involves the collection time corresponding to the value of the indicator in the time series data. Furthermore, the collection time corresponding to the value of the indicator is equivalent to the collection time of the time series data.

[0067] Step 301: The front-end device obtains first time series data.

[0068] In a first possible manner, the front-end device may obtain first time series data from the target device.

[0069] Alternatively, the front-end device can obtain the first time series data from a message middleware, where the message middleware is located between the front-end device and the target device, and the message middleware is, for example, Kafka. For example, the target device can send the collected first time series data to the message middleware, which caches the first time series data. Subsequently, the front-end device reads the first time series data from the message middleware's cache.

[0070] In the second possible way, the front-end device includes a sorting cache, and the sorting cache includes one or more time series data. Exemplarily, the multiple time series data in the sorting cache are sorted in order of collection time from early to late. Exemplarily, after the front-end device obtains any time series data from the target device or the message middleware, it can determine the location in the sorting cache where the newly obtained time series data is stored based on the collection time in the time series data and the collection time in each time series data already in the sorting cache, and then store the newly obtained time series data in the sorting cache based on the location. It can be understood that the earlier the time series data is collected, the earlier it is placed in the sorting cache; and the later the time series data is collected, the later it is placed in the sorting cache. For example, the sorting cache already has time series data a to time series data c, where the collection time of time series data a to time series data c is 0.3 minutes, 0.5 minutes, and 0.7 minutes respectively. Correspondingly, the sorting cache includes time series data a to time series data c in sequence. Furthermore, the front-end device obtains time series data d from the target device or message middleware, and the collection time of time series data d is 0.4 minutes. Then the front-end device can determine that the position of time series data d in the sorting cache is between time series data a and time series data b based on the collection time of time series data d and the collection time of time series data a to time series data c, and then store time series data d between time series data a and time series data b in the sorting cache.

[0071] Exemplarily, the sorting cache may be a memory space reserved in the memory of the front-end device for caching the time series data.

[0072] Furthermore, the front-end device may obtain the first time series data from the head of a sorting buffer (or called a sorting queue) of the front-end device.

[0073] Step 302: The front-end device determines whether the first time series data is sequential data.

[0074] Specifically, when the front-end device determines that the first time series data is sequential data, the following step 303 is executed; when the front-end device determines that the first time series data is out-of-order data (ie, not sequential data), the following step 304 is executed.

[0075] In one possible approach, the front-end device determines whether the first time series data is sequential data. Specifically, the front-end device determines whether the first time series data is the next time series data after the second time series data collected by the target device. Furthermore, when the front-end device determines that the first time series data is the next time series data after the second time series data collected by the target device, the first time series data is determined to be sequential data; and when the front-end device determines that the first time series data is not the next time series data after the second time series data, the first time series data is determined to be out-of-sequence data.

[0076] The second time series data is the time series data most recently stored in the database by the front-end device. For example, the second time series data may be the time series data of the indicator of the target device first acquired by the front-end device.

[0077] In one possible approach, when the front-end device determines whether the first time series data is the next time series data after the second time series data collected by the target device, the front-end device may specifically determine whether the first time series data is the next time series data after the second time series data collected by the target device based on the first collection time, the reference collection time, and the collection rule. The reference collection time is the second collection time in the second time series data. Optionally, the reference collection time is stored in a sorting buffer.

[0078] It is understood that after storing the second time series data in the database, the front-end device can use the second acquisition time in the second time series data as the reference acquisition time. Subsequently, the front-end device obtains the first time series data and determines whether the first time series data is the next time series data of the second time series data based on the first acquisition time and the reference acquisition time and acquisition rule in the first time series data.

[0079] In one possible approach, the collection rule is a rule for collecting time series data from a target device, pre-learned by the front-end device based on multiple historical time series data. The collection times of the multiple historical time series data are all before the second collection time. The method for determining the collection rule by the front-end device can be found in the description of the embodiment related to FIG6 . Optionally, the front-end device can also store the collection rule in a sorting buffer.

[0080] It is noted in advance that the target device can collect time series data periodically or non-periodically. For example, the target device collects time series data at time points such as 00:00:00, 00:02:00, and 00:04:00, that is, the target device collects time series data periodically, and the fixed collection interval is 2 minutes. For another example, the target device collects time series data at time points such as 00:00:02, 00:01:01, and 00:01:59, that is, the target device collects time series data non-periodically. Whether the target device collects time series data periodically is not notified to the front-end device in advance, but is learned by the front-end device based on multiple historical time series data from the target device (see also the description in the relevant embodiment of Figure 6).

[0081] (1) When the target device periodically collects timing data:

[0082] The collection schedule includes fixed collection intervals.

[0083] When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is equal to the fixed acquisition interval, the front-end device determines that the first time series data is the next time series data of the second time series data acquired by the target device. When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is not equal to the fixed acquisition interval, the front-end device determines that the first time series data is not the next time series data of the second time series data acquired by the target device.

[0084] (2) When the target device collects time series data non-periodically:

[0085] It can be considered that the interval length of the collection time of multiple time series data conforms to the exponential distribution.

[0086] In one example, the acquisition rule includes a threshold. When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is less than or equal to the threshold, the front-end device determines that the first time series data is the next time series data of the second time series data acquired by the target device. When the front-end device determines that the difference between the first acquisition time and the reference acquisition time is greater than the threshold, the front-end device determines that the first time series data is not the next time series data of the second time series data acquired by the target device. The threshold is determined based on the average value of the exponential distribution. Exemplarily, the threshold is determined based on the average value of the exponential distribution and the dynamic frequency tolerance. For example, the threshold is equal to the product of the average value of the exponential distribution and the dynamic frequency tolerance.

[0087] In another example, the acquisition rule includes the average value and dynamic frequency tolerance of the exponential distribution. The front-end device can determine the threshold based on the average value and dynamic frequency tolerance of the exponential distribution, and then when it is determined that the difference between the first acquisition time and the benchmark acquisition time is less than or equal to the threshold, it is determined that the first time series data is the next time series data of the second time series data collected by the target device; and when it is determined that the difference between the first acquisition time and the benchmark acquisition time is greater than the threshold, it is determined that the first time series data is the next time series data of the second time series data collected by the target device.

[0088] The dynamic frequency tolerance may be a fixed value set by the front-end device, such as the dynamic frequency tolerance being equal to 1.2.

[0089] Step 303: The front-end device stores the first time series data in a database.

[0090] Specifically, the front-end device stores the first time series data after the second time series data in the database.

[0091] In one possible embodiment, when the front-end device determines, based on the first acquisition time, the benchmark acquisition time, and the acquisition rule, that the first time series data is the next time series data of the second time series data acquired by the target device, the front-end device may update the benchmark acquisition time to the first acquisition time after storing the first time series data in the database.

[0092] In one possible embodiment, the first time series data is time series data obtained by the front-end device from the sorting cache, that is, the first time series data is already stored in the current sorting cache. In this case, the front-end device can also delete the first time series data in the sorting cache.

[0093] In one possible manner, the front-end device may also obtain the time series data with the earliest collection time in the sorting cache (recorded as the third time series data) from the sorting cache, wherein the third collection time of the third time series data is earlier than the collection time of other time series data in the sorting cache. If the front-end device determines that the third time series data is the next time series data of the first time series data collected by the target device, the third time series data is stored in the database and the third time series data in the sorting cache is deleted. Optionally, the front-end device also updates the benchmark collection time to the third collection time. The front-end device again obtains the time series data with the earliest collection time in the current sorting cache (recorded as the fourth time series data) and determines whether the fourth time series data is the next time series data of the third time series data collected by the target device. If the front-end device determines that the third time series data is not the next time series data of the first time series data collected by the target device, the third time series data continues to be stored in the sorting cache.

[0094] It should be noted that the front-end device's processing of the third time series data can be understood by referring to the descriptions in steps 301 to 304, where "first time series data" can be replaced with "third time series data" and "second time series data" can be replaced with "first time series data". Similarly, the front-end device's processing of the fourth time series data can be understood by referring to the descriptions in steps 301 to 304, where "first time series data" can be replaced with "fourth time series data" and "second time series data" can be replaced with "third time series data".

[0095] In step 304 , the front-end device stores the first time series data in a sorting buffer of the front-end device.

[0096] The front-end device obtains the first time series data directly from the target device or message middleware, or obtains the first time series data from the sorting cache. The following describes the different situations:

[0097] (1) When the front-end device obtains the first time series data from the target device or the message middleware:

[0098] The front-end device determines the location of the first time series data in the sorting cache according to the collection time of the first time series data and the collection time of each time series data already in the sorting cache, and then stores the first time series data in the sorting cache according to the location.

[0099] After the front-end device stores the first time series data in the sorting cache, it also obtains new time series data (recorded as the fifth time series data) from the target device or the message middleware. Among them, the fifth time series data includes the fifth acquisition time. When the front-end device determines that the fifth time series data is the next time series data of the second time series data collected by the target device, the fifth time series data is stored in the database. Optionally, the front-end device also updates the benchmark acquisition time to the fifth acquisition time. When the front-end device determines that the fifth time series data is not the next time series data of the second time series data collected by the target device, the fifth time series data is stored in the sorting cache. It should be pointed out that the way the front-end device processes the fifth time series data can be referred to the description in steps 301 to 304, and the "first time series data" can be replaced with "fifth time series data" for understanding.

[0100] The process in Figure 3 can also be considered as follows: if the front-end device determines that the new time series data is sequential data (for example, the next time series data of the time series data most recently stored in the database), then the front-end device stores the new time series data in the database; if it determines that the new time series data is disordered data (that is, not sequential data (for example, not the next time series data of the time series data most recently stored in the database)), then the front-end device stores the new time series data in the sorting cache. In the case where the front-end device stores the new time series data in the database, the front-end device also obtains the time series data with the earliest collection time from the sorting cache, and determines whether the time series data with the earliest collection time in the sorting cache is sequential data. If so, the front-end device stores the time series data with the earliest collection time in the sorting cache in the database, and deletes the time series data that has been stored in the database in the sorting cache. The front-end device then obtains the time series data with the earliest collection time from the sorting cache, and determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and so on, until there is no more time series data in the sorting cache. When the front-end device determines that the time series data with the earliest collection time in the sorting buffer is out-of-order data, the determination is terminated.

[0101] In combination with the processes in FIG3 and FIG4 , an example 1 is provided below:

[0102] The target device periodically collects time series data to obtain time series data A to time series data E. The collection time of time series data A to time series data E is 0.1 minutes, 0.3 minutes, 0.5 minutes, 0.7 minutes, and 0.9 minutes respectively. The fixed collection period is 0.2 minutes.

[0103] The front-end device first obtains time series data A from the target device, determines that time series data A is the first time to be obtained, and then directly stores time series data A in the database, and updates the benchmark acquisition time in the sorting cache to the acquisition time of time series data A (0.1 minute).

[0104] The front-end device continues to obtain time series data B from the target device, and determines that the acquisition time of time series data B (0.3 minutes) and the benchmark acquisition time (0.1 minutes) differ by 0.2 minutes, which is a fixed acquisition cycle. Therefore, it is determined that time series data B is the next time series data of time series data A collected by the target device, and time series data B is stored after time series data A in the database.

[0105] The front-end device continues to obtain time series data D from the target device, and determines that the acquisition time of time series data D (0.7 minutes) differs by 0.4 minutes from the benchmark acquisition time (0.3 minutes), that is, it is not a fixed acquisition cycle. Therefore, it is determined that time series data D is not the next time series data of time series data B collected by the target device, and the time series data D is stored in the sorting cache.

[0106] The front-end device continues to obtain time series data E from the target device, and determines that the acquisition time of time series data E (0.9 minutes) differs from the benchmark acquisition time (0.3 minutes) by 0.6 minutes, that is, it is not a fixed acquisition cycle. Therefore, it is determined that time series data E is not the next time series data of time series data B collected by the target device, and time series data E is stored after time series data D in the sorting cache.

[0107] The front-end device continues to obtain time series data C from the target device, and determines that the difference between the acquisition time of time series data C (0.5 minutes) and the benchmark acquisition time (0.3 minutes) is 0.2 minutes, which is a fixed acquisition cycle. It is determined that time series data C is the next time series data of time series data B collected by the target device, and the time series data C is stored after the time series data B in the database, and the benchmark acquisition time is updated to 0.5 minutes.

[0108] Subsequently, the front-end device obtains the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data D, and determines that the collection time of the time series data D (0.7 minutes) and the benchmark collection time (0.5 minutes) differ by 0.2 minutes, which is a fixed collection cycle. Therefore, it is determined that the time series data D is the next time series data of the time series data C collected by the target device, and the time series data D is stored after the time series data C in the database. The benchmark collection time is updated to 0.7 minutes, and the time series data D in the sorting cache is deleted.

[0109] The front-end device can also obtain the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data E, and determine that the collection time of the time series data E (0.9 min) and the benchmark collection time (0.7 min) are 0.2 min apart, which is a fixed collection cycle. Therefore, it is determined that the time series data E is the next time series data of the time series data D collected by the target device, and the time series data E is stored after the time series data D in the database, the benchmark collection time is updated to 0.9 min, and the time series data E in the sorting cache is deleted.

[0110] (2) When the front-end device obtains the first time series data from the sorting cache:

[0111] The first time series data is the first time series data in the sorting cache. When the front-end device does not obtain new time series data from the target device or message middleware, the front-end device may no longer obtain the first time series data and determine whether the first time series data is the next time series data after the second time series data collected by the target device, thereby avoiding unnecessary operations on the front-end device.

[0112] The front-end device obtains new time series data (i.e., the fifth time series data) from the target device or the message middleware, and stores the fifth time series data before the first time series data in the sorting buffer according to the fifth collection time of the fifth time series data. Subsequently, the front-end device obtains the fifth time series data from the sorting buffer and determines whether the fifth time series data is the next time series data after the second time series data collected by the target device.

[0113] When the front-end device stores the fifth time series data in the database, the front-end device can further obtain the time series data with the earliest collection time in the sorting cache (that is, the first time series data) from the sorting cache, and determine whether the first time series data is the next time series data of the fifth time series data collected by the target device.

[0114] The process in Figure 3 can also be considered as follows: if the front-end device determines that the new time series data is located before the first time series data in the sorting cache (that is, the collection time is earlier than the earliest collection time of the time series data in the current sorting cache), it determines whether the new time series data is sequential data (for example, the next time series data of the time series data most recently stored in the database). If so, the new time series data is stored in the database. When the front-end device stores the new time series data in the database, the front-end device also obtains the time series data with the earliest collection time from the sorting cache, determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and if so, the front-end device stores the time series data with the earliest collection time in the sorting cache in the database and deletes the time series data that has been stored in the database in the sorting cache. The front-end device then obtains the time series data with the earliest collection time from the sorting cache, determines whether the time series data with the earliest collection time in the sorting cache is sequential data, and so on, until there is no more time series data in the sorting cache. If the front-end device determines that the earliest time series data in the sorting cache is out-of-order data (i.e., not sequential data (e.g., not the next time series data after the most recently stored time series data in the database)), the determination ends. For details, see the flowchart of the third data processing method shown in FIG5 .

[0115] Combining the processes in FIG3 and FIG5 , Example 2 is provided below:

[0116] The target device periodically collects time series data to obtain time series data A to time series data E. The collection time of time series data A to time series data E is 0.1 minutes, 0.3 minutes, 0.5 minutes, 0.7 minutes, and 0.9 minutes respectively. The fixed collection period is 0.2 minutes.

[0117] The front-end device first obtains time series data A from the target device and stores it in the sorting cache. Subsequently, the front-end device obtains time series data A from the sorting cache, determines that time series data A is the first time it has been obtained, directly stores time series data A in the database, and updates the baseline acquisition time in the sorting cache to the acquisition time of time series data A (0.1 minute).

[0118] The front-end device continues to obtain time series data B from the target device and stores it in the sorting cache. Subsequently, the front-end device obtains time series data B from the sorting cache and determines that the difference between the acquisition time of time series data B (0.3 minutes) and the benchmark acquisition time (0.1 minutes) is 0.2 minutes, which is a fixed acquisition cycle. Therefore, the front-end device determines that time series data B is the next time series data after time series data A collected by the target device and stores time series data B after time series data A in the database.

[0119] The front-end device continues to obtain time series data D from the target device and stores it in the sorting buffer. The front-end device obtains time series data D from the sorting buffer and determines that the difference between the collection time of time series data D (0.7 minutes) and the reference collection time (0.3 minutes) is 0.4 minutes. This means that they are not in a fixed collection cycle. Therefore, the front-end device determines that time series data D is not the next time series data after time series data B collected by the target device.

[0120] The front-end device continues to obtain time series data E from the target device, and according to the acquisition time of time series data E (0.9 minutes) and the acquisition time of time series data D (0.7 minutes), the time series data E is stored behind the time series data D in the sorting cache.

[0121] The front-end device continues to acquire time series data C from the target device. Based on the acquisition time of time series data C (0.5 minutes), the acquisition time of time series data E (0.9 minutes), and the acquisition time of time series data D (0.7 minutes), time series data C is stored in the sorting cache before data D. Furthermore, the front-end device determines that the acquisition time of time series data C (0.5 minutes) differs by 0.2 minutes from the benchmark acquisition time (0.3 minutes), which is a fixed acquisition cycle. Therefore, time series data C is determined to be the next time series data after time series data B collected by the target device. Time series data C is stored after time series data A in the database, time series data C is deleted from the sorting cache, and the benchmark acquisition time is updated to 0.5 minutes.

[0122] Subsequently, the front-end device obtains the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data D, and determines that the collection time of the time series data D (0.7 minutes) and the benchmark collection time (0.5 minutes) differ by 0.2 minutes, which is a fixed collection cycle. Therefore, it is determined that the time series data D is the next time series data of the time series data C collected by the target device, and the time series data D is stored after the time series data C in the database. The time series data D in the sorting cache is deleted, and the benchmark collection time is updated to 0.7 minutes.

[0123] The front-end device can also obtain the time series data corresponding to the earliest collection time from the sorting cache, that is, the time series data E, and determine that the collection time of the time series data E (0.9 min) and the benchmark collection time (0.7 min) are 0.2 min apart, which is a fixed collection cycle. Therefore, it is determined that the time series data E is the next time series data of the time series data D collected by the target device, and the time series data E is stored after the time series data D in the database. The time series data E in the sorting cache is deleted, and the benchmark collection time is updated to 0.9 min.

[0124] It should be added that, when the front-end device determines that the number of time series data stored in the sorting cache exceeds a preset number, the front-end device may deem that there is a situation where time series data is lost, and thus stores the time series data corresponding to the earliest acquisition time in the sorting cache to the database, and deletes the time series data corresponding to the earliest acquisition time in the sorting cache. Subsequently, the front-end device determines the time series data corresponding to the new earliest acquisition time from the sorting cache, and determines whether the time series data corresponding to the new earliest acquisition time in the sorting cache is sequential data, such as determining whether the time series data corresponding to the new earliest acquisition time in the sorting cache is the next time series data of the time series data most recently stored in the database. If so, the front-end device stores the time series data with the new earliest acquisition time in the sorting cache to the database, and deletes the new time series data in the sorting cache that has been stored in the database. For specific implementation, please refer to the description in the above step 304. Optionally, after the front-end device stores the time series data corresponding to the earliest acquisition time in the sorting cache to the database, it may update the benchmark acquisition time to the earliest acquisition time.

[0125] The preset number may be a number preset by the front-end device, such as 2, 3, 5, etc.

[0126] In combination with the above example 1, assuming that the preset number is 2, after the front-end device stores the time series data E behind the time series data D in the sorting cache, the front-end device can determine that the time series data collected by the target device at the 0.5th minute has been lost, so the time series data D is stored behind the time series data B in the database, and the collection time of the time series data D (0.7th minute) is used as the new benchmark collection time. Subsequently, based on the new benchmark collection time (0.7th minute), the front-end device determines that the collection time of the time series data E (0.9th minute) and the benchmark collection time (0.7th minute) differ by 0.2 minutes, that is, a fixed collection cycle, and thus determines that the time series data E is the next time series data of the time series data D collected by the target device, stores the time series data E behind the time series data D in the database, and updates the benchmark collection time to 0.9th minute.

[0127] In this way, in the event of data loss, the front-end device is prevented from constantly waiting for the lost data and being unable to store the data in the sorting cache into the database in a timely manner, thereby further improving the efficiency of data processing.

[0128] Exemplarily, the front-end device includes a sorting module corresponding to the indicator, which includes a sorting cache and a sorting execution module. The sorting execution module can be used to execute the steps in the embodiments related to Figures 3 to 5 above. Exemplarily, the sorting execution module is used to obtain first time series data, store the first time series data in a database, or store the first time series data in a sorting cache.

[0129] The above describes in detail the implementation method of the front-end device in this application determining whether the first time series data is the next time series data of the second time series data collected by the target device based on the first collection time, the benchmark collection time and the collection rule. The following explains how the front-end device pre-learns the collection rule based on multiple time series data. For details, please refer to Figure 6 as an illustrative flowchart of learning the collection rule.

[0130] In step 601, the front-end device obtains multiple time series data. The time series data here can be considered to be the time series data obtained by the front-end device from the target device or message middleware before the embodiment related to FIG3. That is, the time series data here is historical time series data relative to the embodiment related to FIG3.

[0131] In one possible approach, the front-end device obtains multiple time series data from the target device or message middleware. This multiple time series data is used by the front-end device to learn the collection rules. After learning the collection rules based on the multiple time series data, the front-end device can discard the multiple time series data. For example, each time the front-end device obtains time series data from the target device or message middleware, it caches the time series data in a training cache. That is, the front-end device caches multiple time series data in the training cache.

[0132] In another possible way, after the front-end device obtains multiple time series data from the target device or message middleware, it not only needs to learn the collection rules based on these time series data, but also stores these time series data in the database. In this case, the front-end device can also maintain a benchmark collection time. When the front-end device obtains the collection time of new time series data, it can determine whether the collection time of the new time series data is later than the benchmark collection time. If so, the newly obtained time series data is stored in the database and the benchmark collection time is updated. Otherwise, the newly obtained time series data is directly discarded. Exemplarily, each time the front-end device obtains time series data from the target device or message middleware, it copies the time series data and caches the copied time series data in the training cache. That is, the front-end device caches multiple time series data in the training cache. Optionally, the number of time series data cached in the training cache can be more than the number of time series data stored in the database.

[0133] Exemplarily, the training cache may be a memory space reserved in the memory of the front-end device for caching the time series data.

[0134] When the multiple time series data obtained by the front-end device are disordered data, the front-end device can first sort the multiple time series data according to the acquisition time of the multiple time series data and in the order of acquisition time from early to late to obtain the multiple time series data after sorting, and then execute the following step 602 according to the multiple time series data after sorting.

[0135] In step 602, the front-end device learns the collection time of multiple time series data to obtain a collection rule.

[0136] In one possible approach, the front-end device determines the collection interval between each two adjacent time series data (i.e., the difference between the collection times of each two adjacent time series data) based on the collection time of multiple time series data, and determines the collection rule based on the collection interval between each two adjacent time series data.

[0137] In this application, the time series data can be N, where N is a positive integer. The collection interval between the first time series data and the second time series data in the N time series data after sorting can be recorded as f1, the collection interval between the second time series data and the third time series data can be recorded as f2, the collection interval between the third time series data and the fourth time series data can be recorded as f3, ..., the collection interval between the N-1th time series data and the Nth time series data can be recorded as f N-1 .

[0138] 7 exemplarily shows a flow chart of the front-end device determining whether the collection interval is regular:

[0139] The front-end device determines f1,f2,...,f N-1 When the following formulas 1 and 2 are met, it is determined that the target device collects multiple time series data based on a fixed collection interval. Among them, maxcount returns the value f of the element that appears the most times in the set. i and the number of occurrences n, α is a preset ratio. Further, the front-end device will f i Determine the fixed collection interval and write the fixed collection interval as the collection rule into the sorting cache. i ,n=maxcount({f1,f2,...,f N-1})...Formula 1

[0140] The front-end device determines f1,f2,...,f N-1 If the result does not meet the requirements of formula 1 and formula 2, it is determined that the target device does not collect multiple time series data based on a fixed collection interval. N-1 It conforms to the exponential distribution. The formula of exponential distribution is as follows: Formula 3.

[0141] in, β is the mean of the exponential distribution. p(f;β) is the probability density function of the time interval f.

[0142] The front-end device can verify whether β is credible. When the front-end device determines that β is credible, it can be determined that the collection interval of the target device when collecting multiple time series data is consistent with the exponential distribution. For example, when the front-end device determines , it is determined that β is credible. In one example, the front-end device determines a threshold value based on β and a preset dynamic frequency tolerance δ, for example, the threshold value is equal to β × δ. The front-end device then determines to write the threshold value as the collection rule into the sorting cache. In another example, the front-end device writes β and δ as the collection rule into the sorting cache.

[0143] When the front-end device determines that β is untrustworthy, such as when the front-end device determines When the target device collects multiple time series data, it can be determined that the collection interval when collecting multiple time series data does not conform to the exponential distribution, that is, it is determined that the collection interval when collecting multiple time series data is irregular, and "irregular" is written into the sorting cache. Furthermore, the front-end device can process multiple time series data based on the watermark mechanism. Specifically, the front-end device sets a time window and an allowed delay time. The front-end device not only sorts the multiple time series data whose arrival time is within the time window, but also sorts the time series data whose difference between the collection time and the allowed delay time is less than or equal to the window end time as the time series data within the time window.

[0144] Exemplarily, the front-end device includes a training module corresponding to the indicator, and the training module includes a training cache and a pattern learning module. Exemplarily, the training cache has two states: a triggered state and a non-triggered state. It can be understood that when the training cache is in the non-triggered state, the pattern learning module does not need to store the time series data obtained by the front-end device from the target device or the message middleware in the training cache; when the training cache is in the triggered state, the pattern learning module needs to store the time series data obtained by the front-end device from the target device or the message middleware in the training cache. Furthermore, the triggered state may include a triggered unfull state and a triggered full state, wherein the boundary between the triggered unfull state and the triggered full state may be whether the number of time series data included in the training cache exceeds a preset number, that is, when the number of time series data included in the training cache does not exceed the preset number, the state of the training cache is a triggered unfull state; when the number of time series data included in the training cache exceeds the preset number, the state of the training cache is a triggered full state. When the pattern learning module determines that the training cache is in a triggered and full state, it can retrieve multiple time series data from the training cache and learn based on the acquisition time of the multiple time series data to obtain an acquisition pattern. For example, the pattern learning module can periodically trigger the training cache to be in a triggered state, and after obtaining an acquisition pattern based on the multiple time series data in the current period, instruct the training cache to return to a non-triggered state. For details, see FIG8 , which illustrates an exemplary diagram of a training cache state switch.

[0145] Furthermore, when the training cache is in an untriggered state, the rule learning module determines to transmit the time series data; when the training cache is in a triggered but not full state, the rule learning module copies the time series data and stores it in the training cache; further, when the training cache is in a triggered and full state, the rule learning module obtains multiple time series data from the training cache, learns according to the collection time of the multiple time series data, and obtains the collection rule. For details, please refer to the schematic diagram of another state switching of the training cache shown in Figure 9.

[0146] It should be noted that the embodiments related to Figures 3 to 9 above are all based on a single indicator as an example, that is, based on multiple historical time series data of an indicator, the acquisition rules corresponding to the indicator are learned, and then the multiple time series data of the indicator are sorted according to the acquisition rules corresponding to the indicator. The front-end device in this application can be used to simultaneously process the embodiments related to Figures 3 to 9 above for multiple indicators. Furthermore, the front-end device can distinguish the acquisition rules and time series data of different indicators based on the identification information of the indicator.

[0147] Exemplarily, in a cloud computing scenario, the identification information of an indicator may include one or more of the following: region identifier, availability zone (AZ) identifier, tenant identifier, instance identifier, node identifier, and metric identifier. A region includes one or more availability zones, an availability zone includes one or more tenants, a tenant may deploy one or more instances, an instance may run on one or more nodes, and a node may correspond to one or more indicators. For example, the identification information of an indicator includes Beijing (region identifier) ​​- Xicheng District (availability zone identifier) ​​- Zhang San (tenant identifier) ​​- instance 1 (instance identifier) ​​- server 1 (node ​​identifier) ​​- CPU occupancy (metric identifier), which means that the CPU occupancy of server 1 located in Xicheng District, Beijing, which is applied for by Zhang San to run instance 1, needs to be collected.

[0148] In one possible implementation, the number of indicators of the target device can be K, that is, the front-end device includes K training modules (denoted as training module 1 to training module K) corresponding to the K indicators, and includes K sorting modules (denoted as sorting module 1 to sorting module K) corresponding to the K indicators, where K is an integer greater than 1. Furthermore, for each indicator, in the training module, the rule learning module obtains multiple historical time series data from the training cache, and learns the collection rule of the indicator based on the obtained multiple historical time series data. For details, please refer to Figure 10, a schematic diagram of learning and collecting rules under multiple indicators. In which, the rule learning module can periodically trigger the training cache to be in a triggered state, and the period of triggering the training cache to be in a triggered state by the rule learning modules in different training modules can be the same or different. In the sorting module, the sorting execution module obtains the time series data of the indicator and stores the time series data in the database in order according to the collection rule of the indicator. For details, please refer to Figure 11, a schematic diagram of data sorting under multiple indicators. Optionally, the rule learning module can write the learned collection rule of the indicator into the corresponding sorting cache, and then the sorting execution module can first obtain the collection rule of the indicator from the sorting cache when determining whether to store the time series data of the indicator into the database.

[0149] In this application, the front-end device can continuously obtain time series data from the monitored device. These time series data obtained by the front-end device can be considered as streaming data, that is, the streaming data includes multiple time series data. Figure 12 is a flow chart of the fourth data processing method provided by this application. This flow chart mainly uses the front-end device to obtain streaming data as an example. For any content not described in detail, please refer to the description of the relevant embodiments in Figures 3 to 11. The details are as follows:

[0150] Training module 1 determines, based on historical streaming data 1, that collection rule 1 includes a fixed collection interval of 1 minute, and writes the fixed collection interval of 1 minute into sorting cache 1 of sorting module 1. Furthermore, sorting cache 1 also includes a reference collection time of 10:00:00, indicating that the collection time of the time series data in streaming data 1 last stored in the database was 10:00:00. When sorting module 1 again obtains new time series data in streaming data 1 from the target device or message middleware, it determines whether the new time series data is the next time series data after the time series data last stored in the database based on the fixed collection interval of 1 minute and the reference collection time of 10:00:00.

[0151] The training module 2 determines, based on the historical streaming data 2, that the collection rule 2 includes an exponential distribution average value of 1 min and a dynamic frequency tolerance of 1.1, and writes the exponential distribution average value of 1 min and the dynamic frequency tolerance of 1.1 into the sorting cache 2 of the sorting module 2. Furthermore, the sorting cache 2 also includes a benchmark collection time of 10:00:10, indicating that the collection time of the time series data in the streaming data 2 last stored in the database was 10:00:10. When the sorting module 2 obtains new time series data in the streaming data 2 from the target device or the message middleware again, it determines whether the new time series data is the next time series data of the time series data last stored in the database based on the exponential distribution average value of 1 min, the dynamic frequency tolerance of 1.1, and the benchmark collection time of 10:00:10.

[0152] The training module K determines that the collection of historical streaming data K is irregular and writes this irregularity into the sorting buffer K of the sorting module K. Furthermore, the sorting buffer K includes a time window length of 10 minutes and an allowed delay of 2 minutes. Based on the watermark mechanism, the time window length of 10 minutes, and the allowed delay of 2 minutes, the sorting module K can sequentially store the multiple time series data in the database.

[0153] It should also be added that the time series data of which indicators are collected by the target device is specifically selected by the tenant. In a specific implementation, the tenant can send a collection request to the front-end device, and accordingly, the front-end device receives the collection request from the tenant, and the collection request includes the identification information of the indicator. The front-end device determines the target device from multiple monitored devices based on the identification information of the indicator, and then deploys the monitoring module of the indicator in the target device. The monitoring module is used for the target device to sequentially collect multiple time series data of the indicator of the target device during operation. In a specific implementation, the tenant can log in to the client and send a collection request to the front-end device by operating the client.

[0154] Optionally, the tenant may also send an analysis request to the analysis device. Accordingly, the analysis device receives the analysis request from the tenant, which includes the identification information of the indicator and the analysis period. Based on the identification information and analysis period of the indicator, the analysis device obtains the time series data of the indicator whose collection time is within the analysis period from the database, and then analyzes the operating status of the indicator based on the obtained time series data of the indicator, and displays the obtained operating status of the target device to the tenant. In a specific implementation, the tenant can log in to the client and send an analysis request to the analysis device by operating the client. Accordingly, the analysis device sends the operating status of the target device to the client, and the client displays the operating status to the tenant on the client's display interface.

[0155] In the above technical solution, after the front-end device determines that the first time series data is obtained, it needs to determine whether the first time series data is sequential data. When it is determined that the first time series data is sequential data, the first data will be stored in the database. When it is determined that the first time series data is out of order data, the first time series data will be stored in the sorting cache of the front-end device. In this way, the front-end device is prevented from storing out of order data in the database. When the acquisition time of the newly received data is earlier than the acquisition time of the time series data already stored in the database, the front-end device reads the time series data in the database again and sorts the read data and the newly received data, which affects the database performance. That is, this method can reduce the impact on database performance, reduce resource consumption of data processing, and reduce the problem of write amplification.

[0156] In addition, when the front-end device determines that the first time series data is sequential data, it will store the first data in the database without first storing the first time series in the sorting cache, and sorting the multiple time series data in the sorting cache whose acquisition time is within the preset time window before storing them in the database, thereby solving the problem of the front-end device not storing the time series data in a timely manner, that is, helping to improve the efficiency of data processing.

[0157] The various embodiments described herein may be independent solutions or may be combined according to internal logic, and all of these solutions fall within the scope of protection of this application.

[0158] Based on the above content and the same concept, as shown in Figure 13, a data processing device is exemplarily provided in this application. The device is applied to the front-end device of the data processing system. Exemplarily, the device can be a front-end device or a module in the front-end device.

[0159] The apparatus includes: an acquisition module 1301, a judgment module 1302, and a storage module 1303. The acquisition module 1301 is used to acquire first time series data; the judgment module 1302 is used to determine whether the first time series data is sequential data; and the storage module 1303 is used to store the first time series data in a database if the judgment module 1302 determines that the first time series data is sequential data; or, if the judgment module 1302 determines that the first time series data is out of sequence data, store the first time series data in a sorting cache of a front-end device. In this application, the acquisition module 1301, the judgment module 1302, and the storage module 1303 can be collectively referred to as a sorting execution module.

[0160] In one possible implementation, when determining whether the first time series data is sequential data, the determination module 1302 is specifically used to determine whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

[0161] In one possible implementation, after the storage module 1303 stores the first time series data in the database, the acquisition module 1301 is further used to: obtain third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache that is collected earlier than other time series data; the judgment module 1302 is further used to: determine whether the third time series data is sequential data; the storage module 1303 is further used to: store the third time series data in the database if the judgment module 1302 determines that the third time series data is sequential data.

[0162] In one possible implementation, after storing the first time series data in the sorting cache of the front-end device, the storage module 1303 is further used to: when it is determined that the number of time series data in the sorting cache exceeds a preset number, store the time series data in the sorting cache that is collected earlier than other time series data in the database.

[0163] In one possible implementation, when storing the first time series data in the sorting cache of the front-end device, the storage module 1303 is specifically used to: determine the position for storing the first time series data in the sorting cache based on the first acquisition time of the first time series data and the acquisition time of the existing time series data in the sorting cache; and store the first time series data in the sorting cache based on the position.

[0164] In one possible implementation, when judging whether the first time series data is the next time series data after the second time series data, the judgment module 1302 is specifically used to: judge whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, wherein the collection rule is the rule of collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

[0165] In one possible implementation, the multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; when the judgment module 1302 judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when the difference between the first collection time and the benchmark collection time is equal to the fixed time interval, determine that the first time series data is the next time series data after the second time series data.

[0166] In one possible implementation, the collection time intervals of multiple historical time series data conform to an exponential distribution, and the collection rule includes the average value and dynamic frequency tolerance of the exponential distribution; when the judgment module 1302 judges whether the first time series data is the next time series data after the second time series data based on the benchmark collection time, the collection rule and the first collection time of the first time series data, it is specifically used to: when it is determined that the difference between the first collection time and the benchmark collection time is less than or equal to a threshold, determine that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value and dynamic frequency tolerance of the exponential distribution.

[0167] In a possible implementation, after storing the first time series data in the database, the determination module 1302 is further configured to: update the reference collection time to the first collection time of the first time series data.

[0168] The acquisition module 1301, the determination module 1302, and the storage module 1303 can all be implemented via software or hardware. For example, the implementation of the acquisition module 1301 will be described below using the acquisition module 1301 as an example. Similarly, the implementation of the determination module 1302 and the storage module 1303 can refer to the implementation of the acquisition module 1301.

[0169] As an example of a software functional unit, acquisition module 1301 may include code running on a computing instance. A computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, acquisition module 1301 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, with each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs. Similarly, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Typically, a VPC is set up in one region. To enable cross-region communication between two VPCs in the same region, or between VPCs in different regions, a communication gateway must be set up in each VPC to interconnect the VPCs.

[0170] As an example of a hardware functional unit, acquisition module 1301 may include at least one computing device, such as a server. Alternatively, acquisition module 1301 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0171] The multiple computing devices included in acquisition module 1301 can be distributed in the same region or in different regions. The multiple computing devices included in acquisition module 1301 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in acquisition module 1301 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0172] It should be noted that, in other embodiments, the acquisition module 1301 can be used to execute any step in the method embodiments related to Figures 3 to 12, the judgment module 1302 can be used to execute any step in the method embodiments related to Figures 3 to 12, and the storage module 1303 can be used to execute any step in the method embodiments related to Figures 3 to 12. That is, the steps that the acquisition module 1301, the judgment module 1302, and the storage module 1303 are responsible for implementing can be specified as needed, and the acquisition module 1301, the judgment module 1302, and the storage module 1303 respectively execute different steps in the method embodiments related to Figures 3 to 12 to realize all the functions of the data processing device.

[0173] This application also provides a computing device 100. As shown in Figure 14, computing device 100 includes a processor 101, a memory 102, a communication interface 103, and a bus 104. Processor 101, memory 102, and communication interface 103 communicate with each other via bus 104. Computing device 100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.

[0174] Bus 104 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG14 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 104 may include a path for transmitting information between various components of computing device 100 (e.g., memory 102, processor 101, and communication interface 103).

[0175] The processor 101 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0176] The memory 102 may include a volatile memory, such as a random access memory (RAM). The processor 101 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0177] The memory 102 stores executable program code, and the processor 101 executes the executable program code to implement the functions of the aforementioned acquisition module 1301, judgment module 1302, and storage module 1303, respectively, thereby implementing the method in the method embodiments related to Figures 3 to 12. In other words, the memory 102 stores instructions for executing the method in the method embodiments related to Figures 3 to 12.

[0178] The communication interface 103 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 100 and other devices or a communication network.

[0179] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0180] As shown in Figure 15 , the computing device cluster includes at least one computing device 100. The memory 102 in one or more computing devices 100 in the computing device cluster may store the same instructions for executing the methods in the method embodiments related to Figures 3 to 12 .

[0181] In some possible implementations, the memory 102 of one or more computing devices 100 in the computing device cluster may also respectively store partial instructions for executing the method in the method embodiments related to Figures 3 to 12. In other words, the combination of one or more computing devices 100 can jointly execute the instructions of the method in the method embodiments related to Figures 3 to 12.

[0182] It should be noted that the memory 102 in different computing devices 100 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data processing apparatus. In other words, the instructions stored in the memory 102 in different computing devices 100 can implement the functions of one or more of the acquisition module 1301, the determination module 1302, and the storage module 1303.

[0183] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG16 shows a possible implementation. As shown in FIG16 , two computing devices 100A and 100B are connected via a network. Specifically, the connection to the network is made via a communication interface in each computing device. In this type of possible implementation, the memory 102 in the computing device 100A stores instructions for executing the functions of the acquisition module 1301. At the same time, the memory 102 in the computing device 100B stores instructions for executing the functions of the judgment module 1302 and the storage module 1303.

[0184] The present application also provides a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the methods described in the method embodiments related to Figures 3 to 12 .

[0185] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the method in the method embodiments related to Figures 3 to 12.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a front-end device of a data processing system, the data processing system further comprising a target device and a database, the target device being used to collect a plurality of time series data during the operation of the target device, each time series data comprising the collection time of the time series data, the method comprising: Acquire first time series data; Determining whether the first time series data is sequential data; When it is determined that the first time series data is sequential data, storing the first time series data in the database; Alternatively, when it is determined that the first time series data is out-of-order data, the first time series data is stored in a sorting cache of the front-end device.

2. The method according to claim 1, characterized in that The determining whether the first time series data is sequential data includes: It is determined whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

3. The method according to claim 1 or 2, characterized in that After storing the first time series data in the database, the method further includes: Acquire third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache that is collected earlier than other time series data; Determining whether the third time series data is sequential data; In a case where it is determined that the third time series data is sequential data, the third time series data is stored in the database.

4. The method according to any one of claims 1 to 3, characterized in that After storing the first time series data in the sorting cache of the front-end device, the method further includes: When it is determined that the number of time series data in the sorting cache exceeds a preset number, the time series data in the sorting cache whose collection time is earlier than other time series data is stored in the database.

5. The method according to any one of claims 1 to 4, characterized in that The storing the first time series data into the sorting cache of the front-end device includes: Determine a location in the sorting buffer where to store the first time series data according to a first collection time of the first time series data and a collection time of time series data already in the sorting buffer; According to the position, the first time series data is stored in the sorting cache.

6. The method according to any one of claims 2 to 5, characterized in that The determining whether the first time series data is the next time series data after the second time series data includes: Based on the benchmark collection time, the collection rule and the first collection time of the first time series data, determine whether the first time series data is the next time series data after the second time series data, wherein the collection rule is the rule for collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

7. The method according to claim 6, characterized in that The multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; The determining, according to the reference collection time, the collection rule, and the first collection time of the first time series data, whether the first time series data is the next time series data after the second time series data includes: When the difference between the first acquisition time and the reference acquisition time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data.

8. The method according to claim 6, characterized in that The collection time intervals of the plurality of historical time series data conform to an exponential distribution, and the collection rule includes an average value and a dynamic frequency tolerance of the exponential distribution; The determining, according to the reference collection time, the collection rule, and the first collection time of the first time series data, whether the first time series data is the next time series data after the second time series data includes: When it is determined that the difference between the first acquisition time and the reference acquisition time is less than or equal to a threshold, it is determined that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value of the exponential distribution and the dynamic frequency tolerance.

9. The method according to any one of claims 6 to 8, characterized in that After storing the first time series data in the database, the method further includes: The reference acquisition time is updated to be the first acquisition time of the first time series data.

10. A data processing device, characterized in that: The device is applied to a front-end device of a data processing system, the data processing system further comprising a target device and a database, the target device being used to collect a plurality of time series data during the operation of the target device, each time series data comprising the collection time of the time series data, the device comprising: An acquisition module, used for acquiring first time series data; A judging module, used for judging whether the first time series data is sequential data; A storage module is used to store the first time series data in the database when the judgment module determines that the first time series data is sequential data; or to store the first time series data in the sorting cache of the front-end device when the judgment module determines that the first time series data is out-of-order data.

11. The device according to claim 10, characterized in that When the judgment module judges whether the first time series data is sequential data, it is specifically used to: It is determined whether the first time series data is the next time series data after the second time series data, where the second time series data is the time series data most recently stored in the database by the front-end device.

12. The device according to claim 10 or 11, characterized in that After the storage module stores the first time series data in the database, The acquisition module is further used to: acquire third time series data from the sorting cache of the front-end device, wherein the third time series data is time series data in the sorting cache whose collection time is earlier than other time series data; The judging module is further used to: judge whether the third time series data is sequential data; The storage module is further configured to: when the judgment module determines that the third time series data is sequential data, store the third time series data in the database.

13. The device according to any one of claims 10 to 12, characterized in that After storing the first time series data in the sorting cache of the front-end device, the storage module is further used to: When it is determined that the number of time series data in the sorting cache exceeds a preset number, the time series data in the sorting cache whose collection time is earlier than other time series data is stored in the database.

14. The device according to any one of claims 10 to 13, characterized in that When the storage module stores the first time series data in the sorting cache of the front-end device, the storage module is specifically used to: Determine a location in the sorting buffer where to store the first time series data according to a first collection time of the first time series data and a collection time of time series data already in the sorting buffer; According to the position, the first time series data is stored in the sorting cache.

15. The device according to any one of claims 11 to 14, characterized in that When the judgment module judges whether the first time series data is the next time series data after the second time series data, it is specifically used to: Based on the benchmark collection time, the collection rule and the first collection time of the first time series data, determine whether the first time series data is the next time series data after the second time series data, wherein the collection rule is the rule for collecting time series data of the target device pre-learned based on multiple historical time series data, and the benchmark collection time is the second collection time of the second time series data.

16. The device according to claim 15, characterized in that The multiple historical time series data include multiple time series data collected by the target device at fixed time intervals, and the collection rule includes the fixed time interval; When the judgment module judges whether the first time series data is the next time series data after the second time series data according to the reference collection time, the collection rule and the first collection time of the first time series data, the judgment module is specifically used to: When the difference between the first acquisition time and the reference acquisition time is equal to the fixed time interval, it is determined that the first time series data is the next time series data after the second time series data.

17. The device according to claim 15, characterized in that The collection time intervals of the plurality of historical time series data conform to an exponential distribution, and the collection rule includes an average value and a dynamic frequency tolerance of the exponential distribution; When the judgment module judges whether the first time series data is the next time series data after the second time series data according to the reference collection time, the collection rule and the first collection time of the first time series data, the judgment module is specifically used to: When it is determined that the difference between the first acquisition time and the reference acquisition time is less than or equal to a threshold, it is determined that the first time series data is the next time series data of the second time series data, and the threshold is determined by the average value of the exponential distribution and the dynamic frequency tolerance. of.

18. The device according to any one of claims 15 to 17, characterized in that After storing the first time series data in the database, the judgment module is further configured to: update the reference acquisition time to the first acquisition time of the first time series data.

19. A data processing system, characterized in that: include: Front-end devices, target devices and databases; The target device is used to collect a plurality of time series data during the operation of the target device, each time series data includes a collection time of the time series data; The front-end device is used to execute the method according to any one of claims 1 to 9.

20. A computing device cluster, characterized in that: comprising at least one computing device, each computing device comprising a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

21. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 9.

22. A computer-readable storage medium, characterized in that: The method comprises computer program instructions, and when the computer program instructions are executed by a computing device cluster, the computing device cluster performs the method according to any one of claims 1 to 9.

Images