Time series data processing method and device, equipment, storage medium and program product
By allocating the timing data to multiple sorting processing partitions and calculating the second attribute value, the high cost of sorting processing of massive timing data is solved, and efficient processing of timing data and business accuracy is achieved.
Patent Information
- Application Number
- CN202411303259.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-12-27
AI Technical Summary
The prior art faces high space and time costs when processing timing data, especially when processing massive timing data, which increases the possibility of business errors.
By allocating the time sequence data to multiple sorting processing partitions, and calculating the second attribute value based on the target attribute and weight coefficient, memory usage is reduced, and partition sorting is adopted to reduce time complexity and spatial complexity.
It effectively reduces the time and space costs of time series data processing, ensures the correct order of time series data transmission, and avoids the occurrence of business errors.
Smart Images
Figure CN119201033A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of big data technology, and in particular, relates to a method, device, equipment, storage medium and computer program product for processing time series data. Background Art
[0002] At present, IoT devices generate or receive a large amount of time-stamped data every day. The time-stamped data is called time series data. The large amount of time series data constitutes a time series stream. For users or third-party applications that receive time series streams, the time series streams generated by the device under specific working conditions must follow certain rules, such as transmitting them to users or third-party applications in the order of the time stamps of the time series data from early to late, to ensure accurate business. However, in the case of network fluctuations or service anomalies, the order of the time series data in the time series stream may be confused, resulting in the time series data not conforming to the business order, causing business errors.
[0003] The existing method for processing disordered time series data in time series streams is generally to load the time series stream with disordered time series data into memory for rule processing. However, the huge amount of data in the time series stream leads to a very large memory consumption, and the processing time will also increase exponentially, resulting in high space and time costs. Summary of the invention
[0004] The embodiments of the present application provide a method, apparatus, device, storage medium, and computer program product for processing time series data, which can reduce the space cost and time cost of processing disordered time series data.
[0005] In a first aspect, an embodiment of the present application provides a method for processing time series data, the method comprising:
[0006] Acquire a first time series stream, where the first time series stream includes a plurality of first time series data, and the first time series data includes a plurality of attributes;
[0007] Allocating the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data; the at least one sorting processing partition has a preset arrangement order;
[0008] For each first time series data in each sorting processing partition, a second attribute value corresponding to the first time series data is calculated according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute; the target attribute represents an attribute related to the business among the multiple attributes;
[0009] For each sorting processing partition, sort the plurality of second attribute values in ascending order of the second attribute values to obtain a first sub-sequence flow;
[0010] For each sorting processing partition, the second attribute value in the first sub-time series stream is replaced with the first time series data corresponding to the second attribute value to obtain a second sub-time series stream;
[0011] According to a preset arrangement order, the second sub-time-series flows respectively corresponding to at least one sorting processing partition are sorted to obtain a target time-series flow.
[0012] In a second aspect, an embodiment of the present application provides a time series data processing device, the device comprising:
[0013] An acquisition module, configured to acquire a first time series flow, wherein the first time series flow includes a plurality of first time series data, and the first time series data includes a plurality of attributes;
[0014] An allocation module, configured to allocate the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data; the at least one sorting processing partition has a preset arrangement order;
[0015] a calculation module, configured to calculate, for each first time series data in each sorting processing partition, a second attribute value corresponding to the first time series data according to a first attribute value and a weight coefficient respectively corresponding to at least one target attribute; the target attribute represents an attribute related to the business among the multiple attributes;
[0016] A sorting module, used for sorting the plurality of second attribute values in ascending order of the second attribute values for each sorting processing partition, to obtain a first sub-time sequence flow;
[0017] The sorting module is further used for each sorting processing partition to replace the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value to obtain a second sub-time series flow;
[0018] The sorting module is further used to sort the second sub-time-series flows corresponding to at least one sorting processing partition according to a preset arrangement order to obtain a target time-series flow.
[0019] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the method for processing timing data as in the first aspect is implemented.
[0020] In a fourth aspect, an embodiment of the present application provides a computer storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method for processing time series data as in the first aspect is implemented.
[0021] In a fifth aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for processing time series data as in the first aspect.
[0022] The processing method, device, equipment, storage medium and computer program product of the time series data of the embodiment of the present application can solve the problem of excessive time complexity of time series data sorting processing by allocating the massive first time series data in the first time series stream to at least one sorting processing partition, and divide the first time series data into smaller granularity data for processing in units of partitions, thereby reducing the time cost. For each first time series data in each sorting processing partition, the second attribute value corresponding to the first time series data is calculated according to the first attribute value and weight coefficient corresponding to at least one target attribute. Since the target attribute is an attribute associated with the time series data sending rule, it means that other attributes unrelated to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the capacity of the memory occupied by the time series data, can reduce the memory required for the sorting processing process in the sorting processing partition, and reduce the space cost of the time series data sorting processing. In summary, the embodiment of the present application can reduce the time cost and space cost of processing time series data with disordered order. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0024] Figure 1 It is a flowchart of a method for processing time series data provided by an embodiment of the present application;
[0025] Figure 2 is a flowchart of a method for processing time series data provided by another embodiment of the present application;
[0026] Figure 3 It is a structural schematic diagram of a time series data processing device provided by an embodiment of the present application;
[0027] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION
[0028] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0029] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements.
[0030] At present, IoT devices generate or receive a large amount of data with timestamps every day. The timestamp is the time when the data is generated. This data is called time series data. A large amount of time series data constitutes a time series stream. Under specific working conditions, the time series stream generated by the device must be sent to users or third-party applications according to certain rules to ensure the correctness of the business.
[0031] For example, for IoT devices such as smart sockets, smart lights, and IoT gas meter switches, the timing stream must be sent in ascending order of timestamps to ensure that the devices execute commands in chronological order to ensure the correctness of the business.
[0032] For example, suppose that the gas company sends a command to close the valve of the IoT gas meter because the user has not paid the gas fee. At the same time, the user happens to check that there is a gas arrears fee. After paying the fee, the gas company sends a second command to open the valve. Under normal circumstances, the IoT platform will reach the IoT gas meter in the order of the time when the commands were issued. In this way, the final state of the gas meter is that the valve is open, and the user can use gas normally.
[0033] However, in the case of network fluctuations or service anomalies, the order of time series data in the time series stream may be disordered, causing the order of instructions received by the IoT gas meter to be changed to open the valve first and then close the valve. This will cause the final state of the gas meter to be the valve closed, causing the user to be unable to use gas after paying off the arrears, resulting in a business error.
[0034] The existing methods for dealing with disordered time series data in time series streams often directly load all the massive time series data into memory, and then perform rule processing on the full amount of time series data. When all the time series data in the time series stream are consistent according to the rules, the data is sent to devices, users, third-party applications, etc. However, when the amount of time series data in the time series stream exceeds 100,000, the time required to process the time series data will increase exponentially; at the same time, the memory consumed by 100,000 data can reach hundreds of megabytes. The IoT platform usually generates tens or hundreds of billions of data, and the time complexity and space complexity of rule processing of time series data are both exponential, resulting in high time and space costs.
[0035] In order to solve the problems of the prior art, the embodiments of the present application provide a method, device, equipment, storage medium and computer program product for processing time series data. By allocating the massive first time series data in the first time series stream to at least one sorting processing partition, and dividing the first time series data into smaller granularity data for processing in units of partitions, the problem of excessive time complexity of time series data sorting processing can be solved, and the time cost is reduced. For each first time series data in each sorting processing partition, the second attribute value corresponding to the first time series data is calculated according to the first attribute value and weight coefficient corresponding to at least one target attribute. Since the target attribute is an attribute associated with the time series data sending rule, it means that other attributes unrelated to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the capacity of the memory occupied by the time series data, can reduce the memory required for the sorting processing process in the sorting processing partition, and reduce the space cost of the time series data sorting processing. In summary, the embodiments of the present application can reduce the time cost and space cost of processing time series data with disordered order.
[0036] The following first introduces the method for processing time series data provided in the embodiment of the present application.
[0037] Figure 1 FIG. 1 is a flow chart showing a method for processing time series data provided by an embodiment of the present application. Figure 1 As shown, the method may include the following steps: S101 to S106.
[0038] S101: Acquire a first timing stream.
[0039] In some embodiments, the first time series stream includes a plurality of first time series data, and the first time series data includes a plurality of attributes.
[0040] In some embodiments, the first time series data may be used to represent time series data having at least one target attribute in the attribute, wherein the target attribute may represent the attribute of the time series data involved in the time series flow sending rule. For example, if the rule indicating the time series flow sending is "send the time series flow according to the size of the timestamp", then the target attribute is "timestamp".
[0041] In this step, a first time series stream consisting of first time series data containing at least one target attribute is obtained, that is, time series data directly related to the sending rules of the time series data is obtained as the data object for rule processing in the embodiment of the present application, which can avoid the additional time cost and space cost problems caused by processing time series data unrelated to the processing rules, thereby improving the efficiency of the overall processing of time series data. Among them, rule processing means correcting the order of time series data with disordered order according to the proper sending rules.
[0042] In some embodiments, the attributes of the time series data may include a timestamp, current, voltage, etc., wherein the timestamp is used to indicate the time when the data is generated.
[0043] S102: Allocate a plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data.
[0044] In some embodiments, the sorting processing partition may be used to perform sorting processing on a portion of the time series data in the first time series stream.
[0045] In some embodiments, the above-mentioned sorting processing partitions may have a preset arrangement order, wherein the preset arrangement order is used to orderly merge the time series data in at least one sorting processing partition after the time series data sorting process is completed, so as to obtain an overall orderly time series flow.
[0046] In some embodiments, there may be multiple sorting processing partitions.
[0047] Typically, the order of magnitude of the time series data in the first time series stream is very high, generally exponential. If the entire amount of time series data in the first time series stream is directly sorted, the processing time will increase exponentially, resulting in excessive time complexity for time series data processing.
[0048] This step distributes multiple first time series data in the first time series stream to at least one sorting processing partition, and divides the massive time series data into smaller granularity data for processing in units of partitions, thereby solving the problem of excessive time complexity of time series data sorting processing.
[0049] S103: For each first time series data in each sorting processing partition, calculate a second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute.
[0050] In some embodiments, the first attribute value may be used to represent the value of the target attribute of the time series data in an actual scenario. For example, the target attribute of the time series data generated by an IoT light may include current, wherein the first attribute value corresponding to the current attribute is the current value generated in the actual scenario.
[0051] In some embodiments, the weight coefficient corresponding to the target attribute can be used to indicate the degree of association between the target attribute and the time series data sending rule, and the degree of association can be expressed in the form of the proportion of the target attribute to the time series data sending rule.
[0052] In some embodiments, the second attribute value can be used to sort the timing data according to its sending rule.
[0053] In some embodiments, the second attribute value corresponding to the first time series data is calculated based on the first attribute value and the weight coefficient respectively corresponding to at least one target attribute, and a weighted summation method may be used.
[0054] S104: For each sorting processing partition, sort the plurality of second attribute values in ascending order of the second attribute values to obtain a first sub-sequence flow.
[0055] In this step, since the second attribute value is calculated based on the first attribute value and the weight coefficient corresponding to the target attribute of the time series data, sorting the second attribute values in the order of the second attribute values from small to large can be equivalent to sorting the second attribute values in the order of the time series data sending rules. In this way, the order of the second attribute values in each sorting processing partition complies with the time series data sending rules, so that the second attribute values can be replaced with the corresponding time series data later, and the time series data that complies with the sending rules can be obtained.
[0056] By sorting the second attribute value corresponding to the time series data instead of the time series data itself, since the second attribute value is calculated based on the target attribute that is only associated with the time series data sending rule, other attributes that are not related to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the capacity of the memory occupied by the time series data and can reduce the memory required for the sorting process in the sorting processing partition, thereby reducing the space cost of the time series data sorting process.
[0057] In some embodiments, in order to improve the efficiency of time series data sorting processing, the above-mentioned sorting of multiple second attribute values in ascending order of the second attribute values can be implemented by a shell or quick sort algorithm.
[0058] S105: For each sorting processing partition, replace the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value to obtain a second sub-time series flow.
[0059] After step S104, by replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value, a second sub-time series flow consisting of the overall ordered time series data in each sorting processing partition can be obtained.
[0060] In some embodiments, the second attribute value in the first sub-timing stream is replaced with the first timing data corresponding to the second attribute value. By setting unique identification data that is the same as the second attribute value, a unique correspondence between the timing data and the second attribute value can be established to achieve accurate replacement of the second attribute value with the timing data.
[0061] S106: Sort the second sub-time-series flows corresponding to at least one sorting processing partition respectively according to a preset arrangement order to obtain a target time-series flow.
[0062] In some embodiments, the preset arrangement order is used to orderly merge the second sub-time series flows in at least one sorting processing partition after the time series data sorting process is completed, so as to obtain an overall ordered target time series flow.
[0063] Sending the overall ordered target time sequence stream to IoT devices, users and third-party applications can ensure the accuracy of the business.
[0064] The embodiment of the present application distributes the massive first time series data in the first time series stream to at least one sorting processing partition, and divides the first time series data into smaller granularity data for processing in units of partitions, thereby solving the problem of excessive time complexity of time series data sorting processing and reducing time cost. For each first time series data in each sorting processing partition, the second attribute value corresponding to the first time series data is calculated according to the first attribute value and weight coefficient corresponding to at least one target attribute. Since the target attribute is an attribute associated with the time series data sending rule, it means that other attributes unrelated to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the capacity of the memory occupied by the time series data, can reduce the memory required for the sorting processing process in the sorting processing partition, and reduce the space cost of the time series data sorting processing. In summary, the present embodiment can reduce the time cost and space cost of time series data processing.
[0065] In some embodiments, before step S103: calculating the second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute, the time series data processing method may further include:
[0066] Selecting a target attribute whose relevance to the business meets a first preset threshold from the multiple attributes;
[0067] According to the relevance of the target attribute to the business, the weight coefficient of the target attribute is determined.
[0068] In some embodiments, step S103: calculating the second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to the at least one target attribute may include:
[0069] According to the weight coefficient, the attribute values corresponding to the target attribute are weighted and summed to obtain the second attribute value.
[0070] In some embodiments, the service may be used to represent a rule for sending time series data. The relevance between the target attribute and the service may be used to represent a relevance between the target attribute and the rule for sending time series data.
[0071] The second attribute value corresponding to the first time series data is calculated by using the first attribute value and the weight coefficient corresponding to the target attribute. Since the target attribute is an attribute associated with the time series data sending rule, it means that other attributes irrelevant to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the memory capacity occupied by the time series data and can reduce the memory required for the sorting process in the sorting processing partition, thereby reducing the space cost of the time series data sorting process.
[0072] In some embodiments, the first preset threshold may be manually set. As an example, the first preset threshold may be 90%.
[0073] In some embodiments, the weighted sum of the attribute values corresponding to the target attribute according to the weight coefficient to obtain the second attribute value can be completed by using the following expression:
[0074] G=ai1×x1+ai2×x2+...+aij×xj (1)
[0075] Among them, G represents the second attribute value corresponding to the time series data, aij represents the weight coefficient of the jth attribute of the i-th time series data, and xj represents the first attribute value of the jth attribute of the i-th time series data.
[0076] In some embodiments, before step S105: replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value, the time series data processing method may further include:
[0077] Obtain identification data of the first time series data;
[0078] The identification data and the second attribute value form a key-value pair.
[0079] In some embodiments, step S105: replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value may include:
[0080] Searching for target first time series data corresponding to the second attribute value through the identification data;
[0081] The second attribute value in the first sub-time series flow is replaced with the target first time series data.
[0082] In some embodiments, the identification data is used to establish a unique correspondence between the second attribute value and the target first time series data.
[0083] The identification data and the second attribute value are combined into a key-value pair, and the target first time series data corresponding to the second attribute value is searched through the identification data. This can facilitate replacing the second attribute value with the corresponding time series data according to the identification data in the key-value pair, thereby improving the efficiency of time series data replacement.
[0084] In some embodiments, in order to ensure the unique correspondence between the first time series data and the second attribute value, the identification data of the first time series data may include:
[0085] Input the first time series data into a hash function, and output a hash value corresponding to the first time series data;
[0086] The hash value is used as identification data of the first time series data.
[0087] Among them, the hash function is a function that maps strings to integers and can be used to determine whether two strings are equal. The core idea is to map the input to a range with a small value domain that can be easily compared. Its characteristic is that when the hash values output by the hash function are different, the two strings must be different.
[0088] By inputting each first time series data as a character string into a hash function, a hash value of each first time series data can be obtained. According to the characteristics of the hash function, the hash value of the first time series data can be used as identification data of the time series data.
[0089] In some embodiments, the above hash function may be expressed as follows:
[0090]
[0091] Wherein, L represents the character string length of the first time series data, and s represents the first time series data.
[0092] In some embodiments, the hash value of the first time series data and the second attribute value may be combined into a key-value pair.
[0093] In some embodiments, Figure 2As shown, in order to improve the efficiency of sorting the first time series data, before step S102: allocating the plurality of first time series data to at least one sorting processing partition according to the number of the first time series data, the time series data processing method may further include the following steps:
[0094] S201: Determine the number of sorting processing partitions according to the ratio of the number of first time series data to a second preset threshold;
[0095] S202: Determine a maximum second attribute value and a minimum second attribute value of the first time series data in the first sub-time series flow based on the second attribute value of the first time series data;
[0096] S203: Determine the attribute value interval corresponding to the sorting processing partition according to the number of sorting processing partitions and the maximum second attribute value and the minimum second attribute value;
[0097] S204: Determine a preset arrangement order of the sorting processing partitions according to the attribute value interval.
[0098] In some embodiments, the second preset threshold is used to represent the maximum amount of time series data that can be processed by each sorting processing partition.
[0099] In some embodiments, the attribute value interval is used to determine a preset arrangement order of at least one sorting processing partition.
[0100] As an example, the order of attribute value intervals from small to large may be set as the preset arrangement order of the sorting processing partitions.
[0101] In some embodiments, the second preset threshold can be set manually.
[0102] As an example, the second preset threshold may be 10 5 , the number of sorting processing partitions is determined according to the ratio of the number of the first time series data to the second preset threshold, and the following expression can be used:
[0103]
[0104] Wherein, N represents the number of sorting processing partitions, and X represents the number of first time series data.
[0105] The second preset threshold is controlled at 10 5 It can be ensured that the sorting of time series data in each sorting processing partition is completed within seconds, and the memory usage of time series data in each sorting processing partition does not exceed megabytes.
[0106] In some embodiments, when X>10 10 When the time series data of the first time series flow is divided into M parts, the number of the first time series data in each part is ensured to be less than 10.10 , and then calculate the number of sorting processing partitions using the above expression based on the number of each time series data.
[0107] In some embodiments, the attribute value interval corresponding to the sorting processing partition is determined according to the number of sorting processing partitions and the maximum second attribute value and the minimum second attribute value. The following expression can be used: attribute value interval = (maximum second attribute value - minimum second attribute value) / (number of sorting processing partitions - 1).
[0108] In some embodiments, in order to improve the overall processing efficiency of the first time series data, step S102: allocating the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data may include:
[0109] The first time series data in the first time series stream is evenly distributed to a plurality of sorting processing partitions through an insertion sort algorithm.
[0110] In order to better demonstrate the effect of this embodiment on improving the overall processing efficiency of the first time series data, the expected time generated by the insertion sort algorithm is described below.
[0111] Let ni be a random variable representing the number of time series data in the sorting processing partition B[i]. Since insertion sort runs in quadratic time, the expected time for sorting the time series data in the sorting processing partition B[i] is E[O(ni2)]=O(E[ni2]), and the total expected time for sorting all the time series data in each sorting processing partition is: O(N).
[0112] In order to find O(N), we need to determine the distribution of each random variable ni. Assume there are n sorting processing partitions. Because each sorting processing partition corresponds to l / n of the interval [0, 1), the probability that a time series data falls into the sorting processing partition B[i] is l / n.
[0113] The above situation is similar to the example of throwing a ball: for n balls and n boxes, each throw is independent and falls into any box with probability p = 1 / n. In this way, the probability of ni = k follows the binomial distribution B(k; n, p), with an expected value of E[ni] = np = 1 and a variance of V[ni] = np(1-p) = 1-1 / n.
[0114] Then the random variable for any number of time series data can be expressed as follows:
[0115]
[0116] Applying the above formula (4) to O(N), we can obtain the expected running time of allocating time series data to the sorting processing partitions.
[0117] In some embodiments, step S101: obtaining a first timing flow may include:
[0118] Acquire an initial time series stream, where the initial time series stream includes a plurality of second time series data;
[0119] Filter out a plurality of third time series data including the target attribute from the plurality of second time series data;
[0120] Filter out fourth time series data whose second attribute value belongs to the attribute value interval from the plurality of third time series data;
[0121] A time series stream including a plurality of fourth time series data is determined as a first time series stream.
[0122] In some embodiments, the second time series data is used to represent initial time series data generated by the IoT device in an actual scenario.
[0123] In some embodiments, the target attribute represents an attribute for indicating the timing data involved in the timing flow sending rule.
[0124] Since time series data that does not contain the target attribute will not cause business errors to users and third-party applications that interact with IoT devices, only multiple third time series data that include the target attribute are screened out from multiple second time series data to avoid processing time series data that are not related to the sending rules; at the same time, a discarding strategy is provided to discard the third time series data that does not include the attribute value interval belonging to any sorting processing partition, and to screen out the fourth time series data whose second attribute value belongs to the attribute value interval from multiple third time series data as the data object for sorting processing, which can avoid the additional time cost and space cost problems caused by processing redundant time series data. In this way, the overall processing efficiency of time series data is improved.
[0125] As an example, the attribute value interval of the sorting processing partition is a time interval, which can be the most recent week. If a user needs to pay attention to the power consumption data of the smart socket in the most recent week, the voltage data at a certain time one month ago in the time series data can be discarded.
[0126] In order to more clearly demonstrate the effect of the embodiments of the present application on improving the overall time series data processing efficiency, the time complexity of the time series data processing of the embodiments of the present application is given below.
[0127] The time complexity of sorting the time series data in each sorting partition is ∑O(N i *logN i ), where N i The amount of time series data in the i-th sorting processing partition.
[0128] For the N first time series data to be processed in the first time series stream, M sorting processing partitions can be preset, and the first average time complexity of sorting processing of each sorting processing partition for N / M first time series data is: O(N / M)+O((N / M)*log(N / M)).
[0129] Then the second average time complexity of sorting the first time series data in the first time series flow is: O(N)+O(M*(N / M)*log(N / M)).
[0130] The second average time complexity can be simplified to: O(N+N*logN-N*logM), that is, O(n+k), where k=N*(log N-logM).
[0131] For the existing time series data processing method, the time complexity of sorting is O(n 2 ), obviously, O(n+k) is much faster than O(n 2 ), the time complexity is greatly reduced.
[0132] Figure 3 is a schematic diagram of the structure of a time series data processing device 300 provided in an embodiment of the present application. Figure 3 As shown, the apparatus may include a data acquisition module 310 , an allocation module 320 , a calculation module 330 and a sorting module 340 .
[0133] An acquisition module 310 is used to acquire a first time series flow, where the first time series flow includes a plurality of first time series data, and the first time series data includes a plurality of attributes;
[0134] An allocation module 320, configured to allocate the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data; the at least one sorting processing partition has a preset arrangement order;
[0135] A calculation module 330 is used to calculate, for each first time series data in each sorting processing partition, a second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute; the target attribute represents an attribute related to the business among the multiple attributes;
[0136] A sorting module 340 is used to sort the multiple second attribute values in ascending order of the second attribute values for each sorting processing partition to obtain a first sub-time sequence flow;
[0137] The sorting module 340 is further used for each sorting processing partition to replace the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value to obtain a second sub-time series flow;
[0138] The sorting module 340 is further configured to sort the second sub-time-series flows corresponding to at least one sorting processing partition according to a preset arrangement order to obtain a target time-series flow.
[0139] This embodiment distributes the massive first time series data in the first time series stream to at least one sorting processing partition, and divides the first time series data into smaller granularity data for processing in units of partitions, thereby solving the problem of excessive time complexity of time series data sorting processing and reducing time cost. For each first time series data in each sorting processing partition, the second attribute value corresponding to the first time series data is calculated according to the first attribute value and weight coefficient corresponding to at least one target attribute. Since the target attribute is an attribute associated with the time series data sending rule, it means that other attributes unrelated to the time series data sending rule are removed in the process of calculating the second attribute value, which reduces the capacity of the memory occupied by the time series data, can reduce the memory required for the sorting processing process in the sorting processing partition, and reduce the space cost of the time series data sorting processing. In summary, this embodiment can reduce the time cost and space cost of time series data processing.
[0140] In some embodiments, the time series data processing device 300 further includes a screening module and a determination module;
[0141] A screening module, configured to screen a target attribute whose relevance to the business meets a first preset threshold from a plurality of attributes before calculating a second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to the at least one target attribute;
[0142] A determination module, used to determine a weight coefficient of a target attribute according to the relevance of the target attribute to the business;
[0143] The calculation module 330 is further used to perform weighted summation on the attribute values corresponding to the target attribute according to the weight coefficient to obtain a second attribute value.
[0144] In some embodiments, the time series data processing device 300 further includes a search module and a replacement module;
[0145] The acquisition module 310 is further configured to acquire identification data of the first time series data before replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value; and to form a key-value pair with the identification data and the second attribute value;
[0146] A search module, used to search for target first time series data corresponding to the second attribute value through identification data;
[0147] The replacement module is used to replace the second attribute value in the first sub-time series flow with the target first time series data.
[0148] In some embodiments, the calculation module 330 is further used to input the first time series data into a hash function, output a hash value corresponding to the first time series data, and use the hash value as identification data of the first time series data.
[0149] In some embodiments, the determination module is further used to determine the number of sorting processing partitions according to a ratio of the number of the first time series data to a second preset threshold before allocating the plurality of first time series data to at least one sorting processing partition according to the number of the first time series data;
[0150] The determination module is further used to determine the maximum second attribute value and the minimum second attribute value of the first time series data in the first sub-time series flow based on the second attribute value of the first time series data;
[0151] The determination module is further used to determine the attribute value interval corresponding to the sorting processing partition according to the number of sorting processing partitions and the maximum second attribute value and the minimum second attribute value;
[0152] The determination module is also used to determine the preset arrangement order of the sorting processing partitions according to the attribute value interval.
[0153] In some embodiments, the allocation module 320 is further configured to evenly allocate the first time series data in the first time series stream to a plurality of sorting processing partitions by using an insertion sort algorithm.
[0154] In some embodiments, the acquisition module 310 is further used to acquire an initial time series stream, where the initial time series stream includes a plurality of second time series data;
[0155] The screening module is further used to screen out a plurality of third time series data including the target attribute from the plurality of second time series data;
[0156] The screening module is further used to screen out fourth time series data whose second attribute value belongs to the attribute value interval from the plurality of third time series data;
[0157] The determination module is further used to determine the timing flow including multiple fourth timing data as the first timing flow.
[0158] Figure 4 The various modules in the device shown can be implemented Figure 1 Each step in the above steps can achieve the corresponding technical effect, which will not be repeated here for the sake of brevity.
[0159] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application is shown.
[0160] The electronic device may include a processor 401 and a memory 402 storing computer program instructions.
[0161] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
[0162] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.
[0163] In certain embodiments, the memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.
[0164] The processor 401 implements any one of the time series data processing methods in the above embodiments by reading and executing computer program instructions stored in the memory 402 .
[0165] In one example, the electronic device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.
[0166] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.
[0167] Bus 410 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 410 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
[0168] In addition, in combination with the time series data processing method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any time series data processing method in the above embodiment is implemented.
[0169] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements any one of the time series data processing methods in the above embodiments.
[0170] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0171] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0172] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0173] The above reference is to the flowchart and / or block diagram of the method, device, equipment, storage medium and computer program product according to the embodiment of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0174] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A method for processing time series data, characterized in that: include: Acquire a first time series flow, where the first time series flow includes a plurality of first time series data, and the first time series data includes a plurality of attributes; Allocating the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data; The at least one sorting processing partition has a preset arrangement order; For each of the first time series data in each of the sorting processing partitions, a second attribute value corresponding to the first time series data is calculated according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute; the target attribute represents an attribute related to the business among the multiple attributes; For each of the sorting processing partitions, sorting a plurality of the second attribute values in ascending order of the second attribute values to obtain a first sub-sequence flow; For each of the sorting processing partitions, replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value, to obtain a second sub-time series flow; The second sub-time-series flows respectively corresponding to the at least one sorting processing partition are sorted according to the preset arrangement order to obtain a target time-series flow.
2. The method according to claim 1, characterized in that Before calculating the second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute, the method further includes: Selecting a target attribute whose relevance to the business meets a first preset threshold from the multiple attributes; Determining a weight coefficient of the target attribute according to the relevance between the target attribute and the business; The calculating the second attribute value corresponding to the first time series data according to the first attribute value and the weight coefficient respectively corresponding to at least one target attribute includes: According to the weight coefficient, a weighted sum is performed on the attribute values corresponding to the target attribute to obtain the second attribute value.
3. The method according to claim 1, characterized in that Before replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value, the method further includes: Acquire identification data of the first time series data; Combining the identification data and the second attribute value into a key-value pair; The replacing the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value includes: Searching for target first time series data corresponding to the second attribute value through the identification data; The second attribute value in the first sub-time series flow is replaced with the target first time series data.
4. The method according to claim 3, characterized in that The acquiring identification data of the first time series data includes: Inputting the first time series data into a hash function, and outputting a hash value corresponding to the first time series data; The hash value is used as identification data of the first time series data.
5. The method according to claim 1, characterized in that Before allocating the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data, the method further includes: Determining the number of the sorting processing partitions according to the ratio of the number of the first time series data to a second preset threshold; Determine a maximum second attribute value and a minimum second attribute value of the first time series data in the first sub-time series flow based on the second attribute value of the first time series data; Determining the attribute value interval corresponding to the sorting processing partition according to the number of the sorting processing partitions, the maximum second attribute value, and the minimum second attribute value; A preset arrangement order of the sorting processing partitions is determined according to the attribute value interval.
6. The method according to claim 1, characterized in that The sorting processing partitions are multiple, and the allocating the multiple first time series data to at least one sorting processing partition according to the quantity of the first time series data includes: The first time series data in the first time series flow is evenly distributed to the plurality of sorting processing partitions through an insertion sort algorithm.
7. The method according to claim 5, characterized in that The obtaining of the first timing flow includes: Acquire an initial time series stream, wherein the initial time series stream includes a plurality of second time series data; Filtering out a plurality of third time series data including the target attribute from the plurality of second time series data; Filter out fourth time series data whose second attribute value belongs to the attribute value interval from the plurality of third time series data; A time series stream including a plurality of the fourth time series data is determined as the first time series stream.
8. A time series data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a first time series flow, wherein the first time series flow includes a plurality of first time series data, and the first time series data includes a plurality of attributes; An allocating module, configured to allocate the plurality of first time series data to at least one sorting processing partition according to the quantity of the first time series data; the at least one sorting processing partition has a preset arrangement order; a calculation module, configured to calculate, for each of the first time series data in each of the sorting processing partitions, a second attribute value corresponding to the first time series data according to a first attribute value and a weight coefficient respectively corresponding to at least one target attribute; the target attribute represents an attribute related to the business among the multiple attributes; a sorting module, configured to sort the plurality of second attribute values in ascending order of the second attribute values for each sorting processing partition, to obtain a first sub-time series flow; The sorting module is further configured to replace the second attribute value in the first sub-time series flow with the first time series data corresponding to the second attribute value in each sorting processing partition, so as to obtain a second sub-time series flow; The sorting module is further used to sort the second sub-time-series flows corresponding to the at least one sorting processing partition according to the preset arrangement order to obtain a target time-series flow.
9. An electronic device, characterized in that: The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method for processing time series data as described in any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method for processing time series data according to any one of claims 1 to 7 is implemented.
11. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method for processing time series data as described in any one of claims 1 to 7.