Time series data processing method, apparatus, device, and storage medium

Abstract

The application provides a time series data processing method, device and equipment and a storage medium, and aims at solving the technical problems of high cache redundancy and low service performance in a distributed service cluster network. When a service receives a query request, a key value corresponding to the query request is calculated and obtained, and the query request is forwarded to a corresponding service for processing based on a mapping relationship between the key value and the service in the service cluster. The service determines whether the cache is hit based on the key value. If not, the result data is queried from the background storage service and cached. If the key values of two consecutive query requests are the same and the deviation period is within the reuse interval, the cached result data is completely reused. If the deviation period is greater than the reuse interval, the missing aggregation interval corresponding result data is supplemented and cached, and the result data not meeting the current time condition in the front is also deleted. The application can improve the stability and processing performance of the distributed service.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer and communication, and in particular, to a time series data processing method, device, equipment and storage medium. BACKGROUND

[0002] In software development, data that needs to be processed to obtain is temporarily placed in memory for users to obtain in a fast manner, which becomes data caching. Data caching is beneficial to improve data extraction efficiency and improve service execution and response speed.

[0003] Data caching usually uses a key-value mode, but for time series data of this type, the efficiency of this mode is relatively low, because this mode does not consider the time arrangement characteristics of time series data, which easily causes defects such as cache data redundancy, low hit rate, and huge background data storage service pressure. For example, in a system for monitoring servers in a distributed server cluster system, if the system load of the past day is viewed, user A accesses and caches the past day data a at 2022-5-12 19:08:28, and user B accesses and caches the past day data b at 2022-5-12 19:08:29. The data volume of the two cached data a and b is not only large but also highly coincident, with only one second of data difference, and even only one millisecond in some scenarios. If overlapping data is repeatedly obtained each time, it will bring great pressure to the caching system. SUMMARY

[0004] Therefore, the present application provides a time series data processing method, device, equipment and storage medium, which solves the technical problems of high redundancy of distributed service caching, low service stability and processing performance.

[0005] Based on one aspect of the embodiment of the present application, the present application provides a time series data processing method, which is applied to a service in a distributed service cluster. The method comprises:

[0006] receiving a query request, obtaining a key value according to information carried by the query request, and forwarding the query request to a target service mapped by the key value for processing according to a mapping relationship between the key value and the service;

[0007] In a case where the target service determines that the time condition type of the query request is a relative time period type, if it is determined based on the key value that the local cache is hit and the deviation period between the query initiation time of the current query request and the query initiation time of the cached result data is within the reuse interval (full reuse hit), the result data of the local cache is reused.

[0008] Further, the method for obtaining the key value according to the information carried by the query request is:

[0009] extracting feature information according to the information carried by the query request;

[0010] assembling the feature information into the key value based on a preset data structure;

[0011] The feature information at least includes time condition type, query condition, and aggregation interval, the query condition at least includes table information and time condition, the time condition type includes fixed time period type and relative time period type, and the aggregation interval is used for equal-interval compression of time series data meeting the query condition.

[0012] Further, the method for forwarding the query request to the target service of the key value mapping according to the mapping relationship between the key value and the service is:

[0013] hashing the key value by using a hash function to obtain a hash value, taking the remainder of the number of registered services in the service cluster based on the hash value to obtain a service serial number, and forwarding the query request to the target service corresponding to the service serial number based on the service serial number.

[0014] Further, the method further includes:

[0015] In a case where the target service determines that the time condition type of the query request is the relative time period type, if it is determined based on the key value that the local cache is not hit, the result data meeting the query condition is obtained from the background storage service and returned after being cached locally;

[0016] The result data is stored in a result data object, and the result data object at least includes the key value, the result data, the cache invalidation time, and the cache reuse time; the cache reuse time is determined by the query initiation time of the query request and the reuse interval.

[0017] Further, the method further includes:

[0018] In a case where the target service determines that the time condition type of the query request is the relative time period type, if it is determined based on the key value that the local cache is hit and the deviation period between the query initiation time of the current query request and the query initiation time of the cached result data is greater than the reuse interval (partial hit), the result data corresponding to the aggregation interval that needs to be supplemented is obtained from the background storage service based on the deviation period, the result data not meeting the time condition of the current query request is deleted, the local cache is updated, and the result data meeting the time condition of the current query request is returned to the user end.

[0019] Further, the method further includes:

[0020] In a case where the time condition type of the query request is the fixed time period type, the target service directly returns the cached result data when determining a hit in the local cache based on the key value; and in a case of determining a miss in the local cache, the target service queries the background storage service to obtain the result data, caches and returns the result data.

[0021] Further, the information carried by the query request further includes a cache invalidation time, and the cache invalidation time in the result data object is determined based on the query request initiation time and the cache invalidation time.

[0022] According to another aspect of the embodiments of the present application, the present application further provides a time series data processing device, which is applied to a server in a distributed service cluster, and the server runs a service. The device can be realized in a software, hardware or combination of software and hardware manner. When realized in a software module manner, the program codes of the software module are loaded into a storage medium of a device, and the program codes in the storage medium are read by a processor for execution, so as to realize the functions of the constituent modules in the device. The device comprises:

[0023] a mapping forwarding module, configured to receive a query request, obtain a key value according to the information carried by the query request, and forward the query request to a data processing module of a target service corresponding to the key value according to a mapping relationship between the key value and the service for processing;

[0024] a data processing module, configured to receive a query request, and in a case where the time condition type of the query request is a relative time period type, reuse the cached result data if it is determined that the local cache is hit based on the key value and the deviation time period between the query initiation time of the current query request and the query initiation time of the cached result data is within a reuse interval (full reuse hit).

[0025] Further, the mapping forwarding module extracts feature information according to the information carried by the query request, and assembles the feature information into the key value based on a preset data structure;

[0026] The feature information at least includes a time condition type, a query condition and an aggregation interval, the query condition at least includes table information and a time condition, the time condition type includes a fixed time period type and a relative time period type, and the aggregation interval is used for equal-interval compression of time series data meeting the query condition.

[0027] Further, in a case where the time condition type of the query request is the relative time period type, the data processing module queries the background storage service to obtain result data meeting the query condition and returns the result data after caching if it is determined that the local cache is missed based on the key value;

[0028] The result data is stored in a result data object, and the result data object at least includes a key value, result data, a cache invalidation time, and a cache reuse time; the cache reuse time is determined by a query initiation time of the query request and a reuse interval;

[0029] In a case where the data processing module determines that the time condition type of the query request is the relative time period type, if it is determined that the local cache is hit based on the key value and a deviation time period between the query initiation time of the current query request and the query initiation time of the cached result data is greater than the reuse interval (partial hit), the result data corresponding to the aggregation interval that needs to be supplemented is obtained from the background storage service based on the deviation time period, the result data that does not meet the time condition of the current query request is deleted, the local cache is updated, and the result data that meets the time condition of the current query request is returned to the user end.

[0030] Based on the embodiment of the present application, the present application further provides an electronic device, comprising a processor, a communication interface, a storage medium and a communication bus, wherein the processor, the communication interface and the storage medium complete mutual communication through the communication bus.

[0031] The storage medium is used for storing the computer program.

[0032] The processor is used for executing the computer program stored on the storage medium, and implementing one or more steps of the time series data processing method.

[0033] It should be noted that the method of the present application can be executed by a single device, such as a computer or a server. The method of the present embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the present application, and the multiple devices will interact with each other to jointly complete the method to achieve the purpose of the present application. The multiple devices constitute a mutual command and control relationship. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the prior art. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art according to these drawings of the embodiments of the present application.

[0035] Figure 1 The network environment schematic diagram to which the method provided by an embodiment of the present application is applied;

[0036] Figure 2A time series data processing method implementation step flowchart is provided for an embodiment of the present application;

[0037] Figure 3 An electronic device structure schematic diagram for implementing the time series data processing method provided by the present application is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0038] The terms used in embodiments of the present application are merely for the purpose of describing specific embodiments and are not intended to limit embodiments of the present application. The singular forms "a", "an" and "the" used in embodiments of the present application are also intended to include plural forms, unless the context clearly indicates otherwise. The term "and / or" used in the present application means any or all possible combinations of one or more associated listed items.

[0039] It should be understood that although the terms first, second, third, etc. may be employed in embodiments of the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish one type of information from another type of information. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of embodiments of the present application. In addition, depending on the context, the word "if" used can be interpreted as "when" or "upon" or "in response to determining".

[0040] In some business application systems, the front-end system needs to present time series data of a specific time period (fixed time period) or relative time period (moving time period) to the user. For example, in a network management system, an administrator may often view the server load situation (such as CPU usage, etc.) of a specific date or the past day through the front-end page. The usual way is: the server receives a query request, converts the time condition in the request into an accurate time query range, and in the case that the time query range cannot be accurately matched with the time query range of the previous query request, even if a large amount of data satisfying the time query range condition is cached locally, the server will not use it, but will directly read the original data in the query time range from the back-end database or storage system again, and after processing and caching again, the result data is returned to the front-end system for display. The above processing method will cause a large amount of redundant data to be cached on the business application server side, causing a great waste of cache resources, and the query response efficiency is also not high, especially in a distributed server cluster, since different user requests will be distributed to different servers for processing based on the need for load balancing, caching a large amount of redundant data on different servers will cause greater resource waste, and without a cache invalidation mechanism, a large amount of cache accumulation will eventually cause memory overflow and system crash.

[0041] To address the aforementioned problems, this invention provides a time-series data processing method and an apparatus and device for implementing the method. This method addresses the technical issues of high cache redundancy and low service performance in application scenarios such as data statistics and querying, and data presentation within a distributed service cluster network. The basic idea of ​​this invention is as follows: When a query request is received, a key value corresponding to the query request is calculated. Based on the mapping relationship between the key value and services in the service cluster, the query request is forwarded to the corresponding service for processing. The service determines whether the cache has been hit based on the key value. If not, the result data is queried from the background storage service and cached. If the key values ​​of two consecutive query requests are the same and the deviation time period is within the reuse interval, the cached result data is fully reused. If the deviation time period is greater than the reuse interval, the missing aggregation interval's corresponding result data is supplemented and cached, and the previous result data that does not meet the current time conditions is also deleted. This invention can reduce cache redundancy of result data in a distributed service cluster, improve cache reuse rate, and thus improve the overall performance of caching and services.

[0042] Based on the basic idea of ​​this invention, the specific implementation process of this invention will be described below in conjunction with the accompanying drawings and specific embodiments.

[0043] Figure 1 This is a schematic diagram of the network environment in which the method provided in one embodiment of the present invention is applied. The time-series data caching method provided by the present invention can be applied to single-server application scenarios, and can also be applied to scenarios such as... Figure 1 The distributed cluster application scenario shown is illustrated. Figure 1 In the application scenario examples, service clusters and component clusters are included. Service clusters deploy multiple service processes (referred to as services) with the same business function in a distributed manner. Component clusters provide distributed component deployment. Component clusters are used to provide the underlying storage services and component support required for distributed services. Component clusters and service clusters are typically deployed separately but interconnected via a network.

[0044] A service cluster can include multiple server nodes. Services A, B, and C can be deployed on different servers, all providing the same service. These three services can concurrently process business requests, thereby improving the performance of concurrent processing. The service cluster requires the support of storage service components (such as Elasticsearch) provided by the component cluster, as well as various other components (such as ZooKeeper and Kubernetes).

[0045] Services in a service cluster need to register in the component cluster so that the services can obtain service registration information. For example, services A, B and C are all registered in the Services directory of the Zookeeper component.

[0046] Figure 2The time series data processing method provided by the embodiment of the application realizes a step flowchart, the method is applied to a server in a service cluster, one or more service programs (referred to as services) are run in the server, and the method comprises the following steps:

[0047] Step 201. The user terminal sends a query request to the server, the service receives the query request, obtains a key value according to information carried by the query request, and forwards the query request to a service corresponding to the key value according to a mapping relationship between the key value and multiple services in the service cluster for processing;

[0048] The information carried by the query request includes but is not limited to a type representing a time condition in a query condition (referred to as a time condition type), specific data query and aggregation conditions (referred to as query conditions), and an optional cache invalidation time. The query conditions include but are not limited to a table name in which time series data is stored, a field name of the time series data in the table, a query range of the time series data (referred to as a time range, that is, a time condition in the query conditions), and an additional limiting condition (that is, a condition for limiting the query result range except the time condition, which is optional). The cache invalidation time is used to control the cache time of the result data on the server, which can be specified by the user terminal or configured through a configuration parameter so as to omit the parameter in the query request. In the case where the query request does not carry the cache invalidation time, the default cache invalidation time set by the server can be obtained. The application does not limit the specific form of the information carried by the query request, as long as the feature information of the key value can be obtained according to the carried information, for example, the request time condition type can be obtained by giving a format of the time condition so as to omit the time condition type in the query request.

[0049] The feature information extracted according to the information carried by the query request includes but is not limited to a time condition type, query conditions, and an aggregation interval (a large interval). The time condition type and the query conditions can be directly extracted from the information carried by the query request, and the aggregation interval is obtained by calculation according to a preset aggregation interval generation algorithm. The aggregation interval is used to compress the time series data meeting the query conditions at equal intervals to adapt to the presentation of the query result on the user terminal. For example, the sampling period of the server performance is 1 second, the time condition of the query is to query the CPU usage rate in the past year, and the data amount of the time series data meeting the query conditions and obtained by sampling in the database table will be very large. If the result data meeting the query conditions is completely cached on the server, the server memory resources and network transmission resources consumed will be very large. In addition, the query result data presented on the user terminal usually does not need such fine-grained data, and a statistical chart usually only needs a few hundred or a few thousand points to present the overall change trend of the time series data meeting the conditions, so the application aggregates the time series data meeting the query conditions through the “aggregation time” parameter to reduce the occupation of the server resources and the network bandwidth resources.

[0050] The aggregation interval is calculated based on the point requirement and time condition of the front-end data presentation by the aggregation time generation algorithm. For example, assuming that the change of the server CPU usage rate in the past day is queried, the start time corresponding to the time range is 2022-10-23 15:37:01.000, the end time is 2022-10-24 15:37:01.000, and the time difference obtained by subtracting the start time from the end time is 68400 seconds. According to the experience of the actual application scene, the line chart rendered by the front-end return of 50-200 points of data can meet the demand of trend change data presentation, and can also take into account the performance. Assuming that 100 time series data points are selected as the standard of front-end data presentation, the aggregation interval = 68400 / 100 = 684 seconds, that is, the server takes 684 seconds as the aggregation interval, 684 seconds is close to 10 minutes, and according to the rounding logic, it is more suitable for the production environment to take ten minutes as the aggregation interval (large interval). The time range in the query condition is divided into 100 continuous time periods with the aggregation interval as the unit, then the average value of the time series data falling into each aggregation interval and meeting the query condition is obtained, and then the server packs the key value and the average value of each time series data in the time range into the cache of the server. If the same key value of the relative time period type query request is repeatedly received in a short time, the result data cached last time can be reused.

[0051] The way to obtain the value (i.e., key value) of the query request keyword based on the feature information can be that the extracted feature information (such as time condition type, query condition, aggregation interval) is assembled into a key value based on a preset data structure. For example, the feature information is spliced into a string, and the string is taken as the key value. The feature information can also be transformed uniquely and then spliced, which is not limited by the application.

[0052] The determination method of the mapping relationship between the key value and the plurality of services of the server can be that the mapping relationship between the key value and the services in the service cluster is obtained by a hash-based mapping function.

[0053] Reference Figure 1In the example, assume that the user sends a query request 1 to service B, service B parses the query request and extracts feature information from the query request to obtain a key value key 1, then calculates a hash value of key 1, establishes a mapping relationship between the key value and the service by using the randomness of the hash value, directly uses the obtained hash value to take the remainder of the number N of registered services to obtain a service sequence number of the mapping, and forwards the query request to the service corresponding to the service sequence number for processing. Further transformation can be performed on the hash value and the number of registered services to increase the randomness of the mapping relationship between the hash value and the service, so that the mapping relationship is more uniform. For example, in the following mapping mode, a MurmurHash hash function is used to calculate the hash value of the key value key 1, and then a base HASH BASE is introduced to perform function transformation on the hash value to obtain a more uniform mapping relationship. The function transformation mode is as follows: taking the absolute value of the hash value hashval and then taking the remainder of the base HASH BASE to obtain i, then dividing HASH BASE by the number N of registered services and taking the integer part, and then adding the result of HASH BASE and N taking the remainder to select 0 or 1 to obtain unit; finally, dividing i by unit and taking the integer part to obtain the sequence number of the service to which the key value key 1 is mapped.

[0054] int hashval = Murmur3HashFunction.hash(key1);

[0055] int i = Math.abs(hashval) % HASH_BASE;

[0056] int unit = HASH_BASE / N + (HASH_BASE % N == 0? 0 : 1);

[0057] return i / unit;

[0058] The service can obtain the number of registered services in a manner of pulling a registered service list from a component cluster, or in a manner of configuring by using a global configuration parameter. For example, Figure 1 In the example, service B obtains a registered service list from a zookeeper component in the component cluster, and thus obtains the number N of registered services. After service B obtains the mapped service sequence number 2 by using a hash-based mapping function, service B forwards the query request 1 to service C for processing based on the correspondence between the service sequence number and the registered service (for example, service A corresponds to sequence number 0, service B corresponds to sequence number 1, and service C corresponds to sequence number 2). When service C obtains the query result and caches the query result locally, service C can return the query result to service B in the original path, and service B can return the query result to the user, or service C can directly return the request to the user.

[0059] It can be known through the above steps that the embodiment of the application transmits the query request to different services in the service cluster for parallel processing through key value mapping, thereby improving the processing performance. Since the services with the same key value mapping are fixed and unique, the query request with the same key value will be processed by the fixed service, and the corresponding query result will also be cached on the fixed server. Through the distribution mechanism of the query request, it can be ensured that the cached data with the same key value exists only in one service, thereby reducing data redundancy and avoiding waste of storage space and server resources.

[0060] It should be noted that if the service initially receiving the query request determines that the service processing the query request is itself according to the mapping relationship between the key value and the service, the service can transmit the query request to itself through a loopback port, and the transmission target is itself.

[0061] Step 202. The service processing the query request receives the query request, and determines whether the type of the query request is a fixed time period type according to the time condition in the query condition of the query request. If the type is the fixed time period type, step 210 is performed, and if the type is a relative time period type, step 220 is performed.

[0062] The embodiment of the application divides the query request into a fixed time period type and a relative time period type according to the type of the time condition in the query condition of the query request. The fixed time period type refers to that the time condition given in the query request is a specific time condition, and the relative time period type refers to that the time condition given in the query request is a relative time condition based on the query initiation time.

[0063] Examples of the fixed time period:

[0064] Example 1: Query the CPU usage rate between 00:00:00 on March 1, 2021 and 00:00:00 on April 1, 2021. The query request contains a specific time condition, and the time range is given and will not change dynamically.

[0065] Examples of the relative time period:

[0066] Example 1: Query the server CPU usage rate in the past week. The form of the time condition carried in such a query request can only carry a relative time identifier, for example, identifier 1 represents nearly 1 hour, identifier 2 represents nearly 1 day, identifier 3 represents nearly 1 month, identifier 4 represents nearly 1 year, etc. When processing such a query request, the service can obtain the system time from the local server as the query initiation time, and determine the start and end times of the time condition based on the query initiation time. In another embodiment, the query request can also carry the initiation time of the query request initiated by the user end and carry the relative time or the relative time identifier, and the application does not make specific limitations.

[0067] Step 210. In the case that the request time condition type is the fixed period type, the service determines whether the result data meeting the query condition is cached locally based on the key value, that is, determines whether the cache is hit. If the cache is hit, step 211 is executed; otherwise, step 212 is executed.

[0068] Since the fixed period type query request carries a specific time condition, the time range of the specific time condition does not dynamically change with the query initiation time as the time condition of the relative period type query request, and therefore the result data also does not dynamically change, and it is not necessary to reuse the cached result data as when processing the relative period type query request.

[0069] Step 211. When the fixed period type query request hits the cache, the hit result data is directly returned.

[0070] The result data meeting the query condition obtained from the background and processed by aggregation of the fixed period type query request is cached locally. When the next query request with the same key value is received, the service can directly return the cached result data to the user end. In order to cache the hot result data as much as possible, when the cached result data is hit again before the cache expiration time, the cache expiration time is refreshed at the same time to avoid the cache from being expired too early.

[0071] Step 212. When the fixed period type query request does not hit the cache, the service queries the background storage service to obtain the result data meeting the query condition, stores the key value, result data, and cache expiration time in the result data object in the local cache, and returns the result data to the user end.

[0072] Different request time condition types can use different result data objects for local caching. The result data object of the fixed period type can include but is not limited to the key value, result data, and cache expiration time. Since the fixed period type does not need to reuse the cached result data, it can not include the reuse interval (small interval) attribute field. The result data in the result data object refers to the time series result data processed according to the aggregation interval. The aggregation of the original time series data meeting the query condition in the background database according to the aggregation interval can be performed on the service side of the service cluster after the service obtains the original time series data meeting the query condition, or can be directly performed using the function of the background database itself, and the implementation can be performed on the storage service side of the component cluster, which is not limited in the present application.

[0073] Step 220. In the case that the request time condition type is the relative period type, the service determines whether the result data meeting the query condition is cached locally based on the key value, that is, determines whether the cache is hit. If the cache is not hit, step 221 is executed; if the full reuse is hit, step 222 is executed; and if the partial reuse is hit, step 223 is executed.

[0074] For the query request of the relative time period type, there are three cases for processing when judging whether the result data in the local cache is hit.

[0075] The first case: not hit, that is, there is no cached result data of the relative time period type with the same key value in the local cache; the service needs to query the background storage service to obtain the result data meeting the query condition and cache it locally.

[0076] The second case: full reuse hit, which can be divided into two cases: complete hit and quasi-complete hit. The complete hit means that the cached result data is completely consistent with the time condition of the current query request. For example, the system time extracted by the service is accurate to seconds, the time when the result data is obtained by querying the background last time is 12:00:00, and if the user sends the same query request of the relative time period type again 600 milliseconds after sending the previous query request, the service considers that the two query requests are completely the same, and the cached data result can be directly returned.

[0077] The quasi-complete hit is the case of complete hit cache with negligible error, which means that the key value of the second query request is the same as that of the previous query request, but the deviation of the initiation time of the two query requests, that is, the deviation period, is within the reuse interval. For example, to view the change of the CPU of server W in a day, it is assumed that the aggregation interval determined by the service is 10 minutes, the reuse interval is 1 minute, it is assumed that the user queries the CPU usage in the past day for the first time at 12:00:00, and the service caches the result data of the first query, and the user queries the CPU usage in the past day again at 12:00:30, the time range of the second query is different from that of the first query by 30 seconds, and the difference of 30 seconds is within the reuse interval of 1 minute, so the service considers that the error within the reuse interval is tolerable and negligible, and the service returns the query result of the previous query request to the user end (client front-end interface). The reuse interval in the present application is set to avoid frequent access to the background storage service, so in the case of quasi-complete hit, the service will not query the data from the background storage service.

[0078] The reuse interval can be determined by the service according to the aggregation interval, which can be determined in a fixed mapping manner or by a preset algorithm. For example, a list of correspondence between the aggregation interval and the reuse interval can be stored on the service side, and the service determines the small interval corresponding to the query request according to the preset interval table. For example, in the interval table, the aggregation interval of 1 day corresponds to the reuse interval of 1 minute, the aggregation interval of 1 month corresponds to the reuse interval of 1 hour, etc. For example, the reuse interval can also be determined according to the percentage of the reuse interval in the aggregation interval, for example, the reuse interval can be determined according to one thousandth of the aggregation interval.

[0079] The third case: partial hit, i.e., non-full reuse hit. The partial hit refers to the case that the time range in the time condition in the latter query request deviates from the former query request key value but exceeds the reuse interval. In the case of partial hit, the service needs to query the background storage service to obtain the result data that needs to be supplemented. The query request for the CPU usage rate of server A in the past day can include the following information:

[0080] Request time condition type: relative time period type;

[0081] Query condition: further includes:

[0082] Table information: the table name is tb_Astate, and the time field name is f_CPUrate;

[0083] Time condition: 1, representing the query time range of the past day;

[0084] Limit condition: the region is Beijing

[0085] Cache expiration time: 5 minutes

[0086] Service B receiving the above query request assembles the key value Key1 based on the information carried by the above query request and the aggregation interval, maps Key1 through a hash function to obtain the service mapped by Key1, and sends the query request to service C. Service C receives the query request and matches Key1 in the local cached result data object. Since it is the first query, it does not hit the local cached result data, and service C will organize a SQL query statement based on the query condition. Assuming that the timestamp when the query request is initiated to the background storage service is time1: 2021-06-02 00:00:00:012, then the time range corresponding to the past day is 2021-06-0100:00:00:012-2021-06-02 00:00:00:012. After obtaining the original time series data that meets the query condition from the background storage service, the result data is cached locally based on the aggregation interval. The cached result data object can include but is not limited to the following attribute fields:

[0087] Key value: Key1

[0088] Result data (usually a result data set): includes a plurality of time series data structures organized together in a time sequence relationship, each time series data structure corresponds to an aggregation interval, and the value in the time series data structure is the average value of the original time series data in the corresponding aggregation interval. For example, the aggregation interval is 1 minute, the aggregation interval of the 1st minute corresponds to 60 original time series data records, and the value in the time series data structure corresponding to the 1-minute aggregation interval is the average value of the 60 records. Here is only an example, and other data aggregation methods can also be used, and the present application does not limit the aggregation method.

[0089] Cache invalidation time: time1+cache invalidation time;

[0090] Cache reuse time: time1+reuse interval.

[0091] Wherein, the cache reuse time can be time1+n*reuse interval, and the parameter n can be configured by the system.

[0092] Step 222. In the case of relative time period type and full reuse hit, the hit result data is directly returned to the user terminal.

[0093] In the case of full reuse hit, the result data with the same key value in the cache can be completely reused, and the result data can be directly returned to the user terminal. In order to make the system cache hot data as much as possible, the cache invalidation time in the cache result data object can be refreshed in each full reuse hit, and the original cache invalidation time in the result data object is updated after adding the cache invalidation time to the original cache invalidation time.

[0094] Step 223. In the case of relative time period type and partial hit, the service determines the deviation period of the cache result data according to the time condition in the query request, and obtains the result data corresponding to the aggregation interval that needs to be supplemented based on the deviation period. The result data meeting the time condition of the current query request is returned to the user terminal.

[0095] Based on the foregoing example of user A querying the CPU usage of server A in Beijing data center for nearly one day at time1 for the first time, it is assumed that user A queries the CPU usage of server A in Beijing data center for nearly one day at time2 for the second time. The two query requests are of relative time period type, and the time difference between the two query requests, i.e. the deviation period (time2-time1), satisfies the condition of reuse interval<time2-time1<cache invalidation time. In this case, the service needs to reconstruct the query condition to the background service based on the deviation period to query the original time series data to supplement the result data corresponding to the missing aggregation interval.

[0096] For example, in an embodiment of the present application, the following scheme is adopted: assuming that the time condition of the current query request is "one day", the aggregation interval is 1 minute, the reuse interval is 1 second, the cache expiration time is 3 minutes, the previous query time time1 is 2021-06-02 00:00:00:012, and the current query time time2 is 2021-06-02 00:02:30:012. The deviation period time2-time1=2 minutes and 30 seconds, which satisfies the condition of 1 second<time2-time1<3 minutes, so the service needs to supplement the result data of the deviation period / aggregation interval, that is, the result data corresponding to 2 aggregation intervals after time1. The service needs to query the original time series data between 2021-06-02 00:00:00:013 and time3, that is, 2021-06-02 00:02:00:012, from the background storage service, and then aggregate the result data according to the aggregation interval to add the result data to the tail of the cached result data. After completing the supplement, the result data set that satisfies the relative time "one day" will not contain the result data corresponding to the first two aggregation intervals in the cache, so the result data that does not satisfy the time condition needs to be deleted from the cache. After updating the result data, the cache expiration time in the result data object is updated to time3+cache expiration time, and the cache reuse time is updated to time3+reuse interval. After completing the above update operation, the service can return the cached result data to the user end.

[0097] Another embodiment of the application can also adopt the following scheme: assuming that the time condition of the current query request is "one day" (the start and end time follows the left-closed right-open principle, that is, the start time is included in the time range, and the end time is not included), the aggregation interval is 1 minute, the reuse interval is 1 second, the cache expiration time is 3 minutes, the previous query time time1 is 2021-06-02 00:00:00:012, and the current query time time2 is 2021-06-02 00:02:30:012. The deviation period time2-time1=2min30s, which satisfies the condition of reuse interval 1s<time2-time1<cache expiration time 3min, then the service needs to fill in the result data of the deviation period / aggregation interval, that is, the result data corresponding to 2 aggregation intervals after time1, and update the last cache result, so the service needs to query the original time series data between 2021-06-02 00:00:00:000 and time3, that is, 2021-06-02 00:02:00:012, from the background storage service, and then update and add the result data to the tail of the cache result data after aggregation. After completing the filling, the result data set that meets the relative time "one day" time window will not contain the result data in the cache that does not meet the time condition, at this time, it should be judged according to the situation, if there is no deletion, no (this case is that there is no data in this period of time, this problem often occurs in production, and two new ones should not be added directly after deleting two) then no deletion is needed. After updating the result data, the cache expiration time in the result data object is time3+cache expiration time, then according to the aggregation interval, the lower limit is 2021-06-02 00:02:30, the reuse interval is rounded up to 1s, and finally the cache reuse time is 2021-06-02 00:02:31. After completing the above update operation, the service can return the cache result data to the user end. It should be noted that the start time of the service to the background query data is 2021-06-0200:00:00:000, and this data is the time of the last data of the last cache result value, which is to refresh the value of the last data of the cache result, because the original value is based on the time period 2021-06-02 00:00:00:000-2021-06-02 00:00:00:012, which does not completely contain one minute.On the other hand, if some data is not synchronized in time during the first cache establishment, the result obtained may be inaccurate, for example, during the cache establishment, the latest time of the data stored in the system is 2021-06-01 11:58:00:000, and then during the second cache refresh, the latest data may have been refreshed, if only the data from 2021-06-02 00:00:00:000 to time3, that is, 2021-06-02 00:02:00:012, is supplemented, then the data in the cache from 2021-06-01 11:58:00:000 to 2021-06-02 00:00:00:000 should exist, and will not appear subsequently. Therefore, the query start time specified in the cache update is the time of the last data in the cache result.

[0098] In an embodiment of the present application, the aggregation interval is allowed to be 0, that is, no aggregation of time series data is performed. In the embodiment, the values of the specific aggregation interval and the reuse interval can be determined according to specific business application scenarios, distribution characteristics of time series data, and other factors, and the present application does not make specific limitations.

[0099] Based on the above embodiment, it can be known that, by using the reuse interval, the present application avoids the redundant execution of continuous relative time period type query requests in a short time, and at the same time avoids the long-time non-update of cache data. For the query request with a time deviation between the previous and subsequent queries greater than the reuse interval, the result data of the cache is reused as much as possible, so as to avoid the redundancy of cache data and improve the cache utilization rate. By using the key value mapping distribution of the query request, the flexible configuration of the reuse interval and the aggregation interval, the present application improves the cache efficiency and service processing performance of the service cluster.

[0100] In order to avoid the long-time occupation of the cache space of the service by the result data of the cache which is not used for a long time, the present application further provides a cache invalidation processing mechanism, that is, a separate cache cleaning thread is started in each service, the cache cleaning thread periodically polls the cache invalidation time in each result data object of the cache, and when the system time is greater than the cache invalidation time, the expired result data object is cleared to release the cache in time. The cache invalidation time is allowed to be specified by the user, and in the case where it is not specified, the system default value can also be taken.

[0101] Figure 3An electronic device structure schematic diagram for implementing the time series data processing method provided by the embodiment of the present application is provided, and the device 300 includes a processor 310 such as a central processing unit (CPU), a communication bus 320, a communication interface 340, and a storage medium 330. The processor 310 and the storage medium 330 can communicate with each other through the communication bus 320. The storage medium 330 stores a computer program, and when the computer program is executed by the processor 310, the function of one or more steps in the time series data processing method provided by the present application can be implemented.

[0102] The storage medium can include a random access memory (RAM) and a non-volatile memory (NVM), such as at least one disk memory. In addition, the storage medium can also be at least one storage device located away from the aforementioned processor. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0103] It should be appreciated that embodiments of the present application can be realized by either computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory storage medium. The methods can be implemented in a computer program using standard programming techniques, including non-transitory storage media configured with a computer program, which when loaded into a computer, causes the computer to operate in a specific and predefined manner. Each program can be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language can be a compiled or interpreted language. Furthermore, the programs can be capable of running in stand-alone machines or can be capable of being employed in a distributed computing environment, with programs stored and executed in local and remote computer storage media as needed. It will be appreciated that the operations of the processes described herein can be performed in an arbitrary sequence based on the implementation, unless otherwise specified or clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) can be performed under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications) to perform operations described herein, and can be implemented using hardware or combinations of hardware and software.

[0104] Further, the methods can be implemented in any suitable type of computing platform operably connected to, including but not limited to, a personal computer, mini-computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platforms, or in communication with charged particle tools or other imaging devices, and the like. Aspects of the present application can be implemented in machine readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optical read and / or write storage media, RAM, ROM, and the like, such that it can be read by a programmable computer and used to configure and operate the computer to perform the processes described herein when the storage medium or device is read by the computer. Furthermore, the machine readable code, or portions thereof, can be transmitted over wired or wireless networks. The present application includes these and other different types of non-transitory computer readable storage media when such media include instructions or programs implementing the steps described above in conjunction with a microprocessor or other data processor. The present application also includes the computer itself when programmed according to the methods and techniques described herein.

[0105] The above description is merely illustrative of the embodiments of the present application and is not in any way limiting. The present application can be modified and changed by various alterations and changes made in the spirit and principles of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application should be included in the scope of the present application.

Claims

1. A time series data processing method, characterized in that, This method is applied to services in a distributed service cluster, and includes: Receive query requests and extract feature information based on the information carried in the query requests; The feature information is used to assemble key values ​​based on a preset data structure; the feature information includes at least: time condition type, query condition, and aggregation interval; the query condition includes at least table information and time condition; the time condition type includes fixed time period type and relative time period type; the aggregation interval is used to compress time series data that meets the query condition at equal intervals; The key value is hashed using a hash function to obtain a hash value. The hash value is then moduloed by the number of services registered in the service cluster to obtain the service sequence number. Based on the service sequence number, the query request is forwarded to the target service corresponding to the service sequence number for processing. If the target service determines that the time condition type of the query request is a relative time period type, and if the local cache is hit based on the key value and the time period between the query initiation time of the current query request and the query initiation time of the cached result data is within the reuse interval, the local cached result data is reused.

2. The method according to claim 1, characterized in that, The method further includes: If the target service determines that the time condition of the query request is a relative time period type, and if the local cache is not hit based on the key value, the backend storage service is queried to obtain the result data that meets the query conditions and the result data is returned after being cached locally. The result data is stored in a result data object, which includes at least: key value, result data, cache expiration time, and cache reuse time; the cache reuse time is determined by the query initiation time and reuse interval of the query request.

3. The method according to claim 1, characterized in that, The method further includes: If the target service determines that the time condition of the query request is a relative time period type, and if the local cache is hit based on the key value and the time difference between the query initiation time of the current query request and the query initiation time of the cached result data is greater than the reuse interval, the backend storage service is queried based on the time difference to obtain the result data corresponding to the aggregation interval that needs to be filled, and the result data that does not meet the time condition of the current query request is deleted. The local cache is then updated, and the result data that meets the time condition of the current query request is returned to the user.

4. The method according to claim 1, characterized in that, The method further includes: When the time condition type of the query request is a fixed time period type, if the target service determines that the local cache has been hit based on the key value, it directly returns the cached result data; if the local cache has not been hit, it queries the backend storage service to obtain the result data, caches it, and returns the result data.

5. The method according to claim 2, characterized in that, The query request also includes a cache expiration time, and the cache expiration time in the result data object is determined based on the query request initiation time and the cache expiration time.

6. A time series data processing apparatus, characterized in that, This device is applied to a server in a distributed service cluster, the server running services, the services including: The mapping and forwarding module receives query requests, extracts feature information based on the information carried in the query request, and assembles key values ​​using the feature information based on a preset data structure. The feature information includes at least: time condition type, query condition, and aggregation interval. The query condition includes at least table information and time condition. The time condition type includes fixed time period type and relative time period type. The aggregation interval is used to compress time series data that meets the query condition at equal intervals. The key values ​​are hashed using a hash function to obtain a hash value. The hash value is then moduloed by the number of services registered in the service cluster to obtain a service sequence number. Based on the service sequence number, the query request is forwarded to the target service corresponding to the service sequence number for processing. The data processing module is used to receive query requests. If the time condition type of the query request is determined to be a relative time period type, and if the local cache is hit based on the key value and the time period between the query initiation time of the current query request and the query initiation time of the cached result data is within the reuse interval, the local cached result data is reused.

7. The apparatus according to claim 6, characterized in that, If the data processing module determines that the time condition of the query request is a relative time period type, and if the local cache is not hit based on the key value, the module queries the background storage service to obtain the result data that meets the query conditions and returns the result data after caching it locally. The result data is stored in a result data object, which includes at least: key value, result data, cache expiration time, and cache reuse time; the cache reuse time is determined by the query initiation time of the query request and the reuse interval. When the data processing module determines that the time condition of the query request is a relative time period type, if the local cache is hit based on the key value and the deviation time period between the query initiation time of the current query request and the query initiation time of the cached result data is greater than the reuse interval, the module queries the background storage service based on the deviation time period to obtain the result data corresponding to the aggregation interval that needs to be filled, deletes the result data that does not meet the time condition of the current query request, updates the local cache, and returns the result data that meets the time condition of the current query request to the user.

8. An electronic device, characterized in that, It includes a processor, a communication interface, a storage medium, and a communication bus, wherein the processor, the communication interface, and the storage medium communicate with each other through the communication bus; Storage medium used to store computer programs; A processor, when executing a computer program stored on a storage medium, performs the method steps of any one of claims 1-5.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the method as described in any one of claims 1 to 5.

Images