Time series-based data processing method, device, equipment and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a time series-based data processing method, device, equipment and storage medium, the method comprises the following steps: determining a period length, when current data does not have periodicity, the period length is infinite, based on the data length of batch processing corresponding to a target time window, data processing delay and data generation interval, adjusting the memory size to realize time series-based data processing; when the current data has periodicity, based on the period length, adjusting the memory size to realize time series-based data processing. In the foregoing manner, the application adaptively identifies the period of the current data; the hash value of the current data is calculated; based on the hash value of the current data and the hash values of the previous n windows, the period length of the data is determined, and when the data does not have periodic characteristics, the period is infinite. Through pre-allocation of the memory required for data processing, the frequent creation of new data and the memory recovery of garbage data are reduced, and the performance and reliability of data processing in the Internet of Things are improved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a time-series-based data processing method, apparatus, device, and storage medium. Background Technology

[0002] The Internet of Things (IoT), or "Internet of Everything," is a network extending and expanding upon the internet. It combines various information sensing devices with the network to form a vast network, enabling interconnection and interoperability between people, machines, and things anytime, anywhere. Data generated in real time by various devices will be reliably transmitted and processed in real time.

[0003] High-frequency trading (HF / HF) refers to algorithmic, high-frequency trading in financial markets, utilizing market change information that is difficult for humans to capture. HF / HF differs from manual trading, where traditional manual trading can only issue a few trading orders per second. HF / HF, through algorithmic trading, accesses market data with low latency, processing real-time market data and generating trading orders at the millisecond level, theoretically capable of completing hundreds of trading orders per second. Real-time market data is generated at the millisecond level and is time-series, requiring a high-performance system for real-time processing and computation, placing high demands on both computational latency and memory usage. Therefore, improving the performance and reliability of data processing in the Internet of Things (IoT) has become a pressing technical challenge. Summary of the Invention

[0004] This application provides a time-series-based data processing method, apparatus, device, and storage medium to improve the reliability of data processing in the Internet of Things.

[0005] In a first aspect, this application provides a time-series-based data processing method, the method comprising:

[0006] Obtain the current data at the target time and obtain the hash value of the current data;

[0007] Based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time, the period length L of the current data is determined;

[0008] When the current data does not have a periodicity, the period length is infinite;

[0009] Based on the data length of the corresponding batch processing within the target time window, the data processing delay, and the data generation interval, the memory size is adjusted to achieve processing based on time series data;

[0010] When the current data has a periodicity, the memory size is adjusted based on the period length to achieve processing based on time series data.

[0011] Furthermore, when the current data does not have a periodicity, the period length is infinite. Based on the batch processing data length corresponding to the target time window, the memory size is adjusted to achieve time-series-based data processing, including:

[0012] When the current data does not have periodicity, the time series data is written to a continuous target memory segment based on a circular queue, and the new data generated by the time series overwrites the historical data in the circular queue.

[0013] Furthermore, when the current data exhibits periodicity, the memory size is adjusted based on the period length to achieve time-series data processing, including:

[0014] The current confidence value is determined based on a preset formula and the number of repetitions.

[0015] The confidence values ​​are compared with the preset confidence threshold, and the memory length is adjusted according to the comparison.

[0016] Further, comparing the confidence values ​​with a preset confidence threshold and adjusting the memory length based on the comparison includes:

[0017] The number of data points is compared with the target period length, and the memory length is adjusted based on the larger of the number of data points and the target period length.

[0018] Further, before comparing the number of data points with the target period length, and adjusting the memory length based on the larger of the number of data points and the target period length, the process includes:

[0019] When the current data does not exhibit periodicity, the number of stored data items is calculated according to a preset formula, which is as follows:

[0020]

[0021] Where m is the number of stored data, c is the data processing and calculation delay, n is the number of data to be processed in the batch corresponding to the target time window, w is the data generation interval, and math.cealing is integer rounding up;

[0022] When the current data has a periodicity, the number of data points is equal to the period length.

[0023] Furthermore, when the current data exhibits periodicity, the memory size is adjusted based on the period length to implement time-series data processing, including:

[0024] When processing data from at least one time point prior to the current time, the number of data items to be stored is determined based on a preset formula;

[0025] Based on the number of stored data items and the preset continuous length, the target memory data is written into memory.

[0026] Further, based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time, the period length of the current data is determined, including:

[0027] When it is determined that the current hash value is equal to the historical hash value, the target time window and the N time windows before the target time are obtained;

[0028] The period length of the current data is determined based on the target time window and the N time windows preceding the target time.

[0029] Secondly, this application also provides a time-series-based data processing apparatus, the apparatus comprising:

[0030] The hash value acquisition module is used to acquire the current data at the target time and acquire the hash value of the current data;

[0031] The period length determination module is used to determine the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time.

[0032] The infinite period module is used when the current data does not have periodicity, and the period length is infinite.

[0033] The first processing module is used to adjust the memory size based on the data length of the corresponding batch processing of the target time window, the data processing delay, and the data generation interval to achieve processing based on time series data.

[0034] The second processing module is used to adjust the memory size based on the period length when the current data has periodicity in order to achieve processing based on time series data.

[0035] Thirdly, this application also provides an apparatus comprising a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the time-series-based data processing method as described above.

[0036] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the time-series-based data processing method described above.

[0037] This application discloses a time-series-based data processing method, apparatus, device, and storage medium. The time-series-based data processing method includes acquiring current data at a target time and acquiring the hash value of the current data; determining the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the previous N time windows; when the current data does not have periodicity, the period length is infinite; adjusting the memory size based on the batch processing data length corresponding to the target time window to achieve time-series-based data processing; when the current data has periodicity, adjusting the memory size based on the period length to achieve time-series-based data processing. Through the above methods, this application adaptively identifies the period length of the current data, specifically, when the data does not have periodicity, the period length is infinite; calculates the hash value of the current data; and automatically determines the period length of the data based on the hash value of the current data and the hash values ​​of the data from the previous n windows, thereby improving the performance and reliability of data processing in the Internet of Things. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic flowchart of a time-series-based data processing method provided for the first embodiment of this application;

[0040] Figure 2 A schematic flowchart of a time-series-based data processing method provided for a second embodiment of this application;

[0041] Figure 3 A schematic flowchart illustrating a time-series-based data processing method provided in the third embodiment of this application;

[0042] Figure 4 A schematic block diagram of a time-series-based data processing apparatus provided for embodiments of this application;

[0043] Figure 5 A schematic block diagram of the structure of a device provided for an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0046] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0047] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0048] Embodiments of this application provide a time-series-based data processing method, apparatus, device, and storage medium. The time-series-based data processing method can be applied to a server, determining the periodicity of the data by calculating the hash values ​​corresponding to the data at the current time and adjacent times, and adjusting the memory length according to the period to improve the reliability of data processing in the Internet of Things (IoT). The server can be a standalone server or a server cluster.

[0049] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0050] Please see Figure 1 , Figure 1 This is a schematic flowchart illustrating a time-series-based data processing method provided in the first embodiment of this application. This time-series-based data processing method can be applied to a server to determine the periodicity of data by calculating the hash values ​​corresponding to the data at the current time and adjacent times, and to adjust the memory length according to the period, thereby improving the reliability of data processing in the Internet of Things.

[0051] like Figure 1 As shown, the time series-based data processing method specifically includes steps S10 to S40.

[0052] Step S10: Obtain the current data at the target time and obtain the hash value of the current data;

[0053] Step S20: Determine the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time.

[0054] Step S30: When the current data does not have periodicity, the period length is infinite;

[0055] Step S40: Based on the data length of the corresponding batch processing in the target time window, the data processing delay and the data generation interval, adjust the memory size to achieve processing based on time series data;

[0056] Step S50: When the current data has periodicity, adjust the memory size based on the period length to achieve processing based on time series data.

[0057] In a specific embodiment, high-frequency time series data has the following characteristics:

[0058] 1. Data is generated at a high frequency, requiring fast and low-latency processing of the current window of data.

[0059] 2. The data volume is large; for example, 1MB of data is generated every 100ms, resulting in 600MB of data per minute. The data will consume and fill up the memory extremely quickly.

[0060] 3. The data lifecycle is short. Calculations at time t typically use data from the most recent time tn to time t. After n time windows, the data becomes garbage data that can be cleaned up. For example, when n=3, data generated every 100ms will become garbage data 300ms after its generation and can be cleaned up from memory.

[0061] In modern high-level computer programming languages, such as Java and Go, memory management is handled by the virtual machine. Memory creation, reclamation, and cleanup are all automatically completed by the virtual machine. The problem encountered in processing high-frequency time-series data streams is that the high frequency of new data generation leads to frequent memory creation and reclamation, which significantly impacts computational performance.

[0062] Data generation -> Allocate new memory -> Store data in memory -> Calculate data -> Memory reaches its limit -> Virtual machine triggers garbage collection -> Clean up and release memory containing historical data.

[0063] Frequent memory creation and virtual machine GC (garbage collection, which cleans up and releases garbage data in memory) will seriously affect system performance and data processing latency.

[0064] This embodiment discloses a time-series-based data processing method, apparatus, device, and storage medium. The time-series-based data processing method includes acquiring current data at a target time and acquiring the hash value of the current data; determining the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the previous N time windows; when the current data does not have periodicity, the period length is infinite; adjusting the memory size based on the batch processing data length corresponding to the target time window to achieve time-series-based data processing; when the current data has periodicity, adjusting the memory size based on the period length to achieve time-series-based data processing. Through the above method, this application adaptively identifies the period length of the current data, specifically, when the data does not have periodicity, the period length is infinite; calculates the hash value of the current data; and automatically determines the period length of the data based on the hash value of the current data and the hash values ​​of the data from the previous n windows, thereby improving the performance and reliability of data processing in the Internet of Things.

[0065] based on Figure 1 In the illustrated embodiment, step S40 is followed by:

[0066] When the current data does not have periodicity, the time series data is written to a continuous target memory segment based on a circular queue, and the new data generated by the time series overwrites the historical data in the circular queue.

[0067] Please see Figure 2 , Figure 2 This is a schematic flowchart illustrating a time-series-based data processing method provided in the second embodiment of this application. This time-series-based data processing method can be applied to a server to determine the periodicity of data by calculating the hash values ​​corresponding to the data at the current time and adjacent times, and to adjust the memory length according to the period, thereby improving the reliability of data processing in the Internet of Things.

[0068] based on Figure 1 The embodiment shown in this embodiment is as follows: Figure 2 As shown, step S50 specifically includes steps S501 to S502.

[0069] Step S501: Determine the current confidence value based on the preset formula and the number of repetitions;

[0070] Step S502: Compare the relationship between the confidence values ​​and the preset confidence threshold, and adjust the memory length according to the relationship.

[0071] In a specific embodiment, we assume that the maximum period length we want to discover is 1.

[0072] (1) For the data generated at time t, take the nearest t-0, t-1, ... t-q+1 data respectively, and calculate the hash value of each data group, denoted as h(t,q), 0<=q<=1.

[0073] (2) Compare each set of h(t,q) with h(t-1,q). h(t-1,q) represents the hash value calculated at time t-1 of the previous time window. When a set of h(t,q) is equal to h(t-1,q), it indicates that a recurring periodic data has been detected, with a period length of q. The number of detected periods d(q) is incremented by 1. (d(0), d(1)...d(l) represent the number of times a period was successfully detected, with an initial value of 0)

[0074] (3) If d(q) >= w, where w is the set confidence threshold for periodic detection, and w >= 1, we are certain that the data is likely to have a data period of length q in the future. Therefore, we readjust the memory length to max(m,q), (m is the same as in optimization scheme 1). For the data arriving at time t+1, we compare the hash value of a single data with the next data in the period. If they are equal, it means that the new data maintains the period, and we can directly use the current data for calculation. If the hash values ​​are inconsistent, it means that the periodic pattern has been broken, and we return to step (1) to start detection again.

[0075] This embodiment discloses a time-series-based data processing method, apparatus, device, and storage medium. The time-series-based data processing method includes determining a current confidence value based on a preset formula and the number of repetitions; comparing the confidence values ​​with a preset confidence threshold; and adjusting the memory length according to the comparison. Through this method, this application determines the periodicity of data by calculating the hash values ​​corresponding to the data at the current time and adjacent times, and adjusts the memory length according to the period, thereby improving the reliability of data processing in the Internet of Things.

[0076] based on Figure 1 In the illustrated embodiment, step S502 specifically includes:

[0077] The number of data points is compared with the target period length, and the memory length is adjusted based on the larger of the number of data points and the target period length.

[0078] based on Figure 2 In the illustrated embodiment, before step S502, the following steps are included:

[0079] When the current data does not exhibit periodicity, the number of stored data items is calculated according to a preset formula, which is as follows:

[0080]

[0081] Where m is the number of stored data, c is the data processing and calculation delay, n is the number of data to be processed in the batch corresponding to the target time window, w is the data generation interval, and math.cealing is integer rounding up;

[0082] When the current data has a periodicity, the number of data points is equal to the period length.

[0083] Specifically, when data is generated at time t, suppose we need to batch process and calculate the data from tn to t. In particular, when n = 0, it means that only the currently generated data is processed and calculated, and data from previous time windows is not needed. Assume the delay for each batch of data processing is c, and the data generation interval is w.

[0084] At system startup, we allocate contiguous memory to create a circular queue for storing data. The allocated memory size is such that it can store at least m data items, where m = math.cealing(n + c / w) (math.cealing represents rounding up an integer).

[0085] A circular queue reuses pre-allocated contiguous memory, avoiding frequent memory creation and cleanup. When the number of data items in memory reaches m, the (m+1)th data item will be stored starting from the beginning of memory.

[0086] Furthermore, such as Figure 3 As shown, Figure 3 This is a schematic flowchart of a time-series-based data processing method provided in the third embodiment of this application.

[0087] Step S50 further includes steps S503 to S504.

[0088] Step S503: When processing data from at least one time point prior to the current time, determine the number of data items to be stored based on a preset formula;

[0089] Step S504: Based on the number of stored data and the preset continuous length, write the target memory data into memory.

[0090] This embodiment discloses a time-series-based data processing method, apparatus, device, and storage medium. The time-series-based data processing method includes determining the number of stored data items based on a preset formula when processing data from at least one time point prior to the current time point; and writing the target memory data into memory based on the number of stored data items and a preset continuous length. Through this method, when the periodicity of the data is determined, this application adjusts the memory length according to the period length characteristics of the data and the data length of each time-point window batch, thereby improving the performance and reliability of data processing in the Internet of Things.

[0091] Based on all the above embodiments, in this embodiment, step S20 specifically includes:

[0092] When it is determined that the current hash value is equal to the historical hash value, the target time window and the N time windows before the target time are obtained;

[0093] The period length of the current data is determined based on the target time window and the N time windows preceding the target time.

[0094] Please see Figure 4 , Figure 4 This application provides a schematic block diagram of a time-series-based data processing apparatus for executing the aforementioned time-series-based data processing method. The time-series-based data processing apparatus can be configured on a server.

[0095] like Figure 4 As shown, the time-series-based data processing device 400 includes:

[0096] The hash value acquisition module 410 is used to acquire the current data at the target time and acquire the hash value of the current data;

[0097] The period length determination module 420 is used to determine the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the N time windows before the target time.

[0098] The infinite period module 430 is used to make the period length infinite when the current data does not have periodicity.

[0099] The first processing module 440 is used to adjust the memory size based on the data length of the corresponding batch processing of the target time window, the data processing delay and the data generation interval to realize the processing of time series data.

[0100] The second processing module 450 is used to adjust the memory size based on the period length when the current data has periodicity in order to realize the processing of time series data.

[0101] Furthermore, the time-series-based data processing device also includes a circular queue module, specifically comprising:

[0102] The circular queue module is used to write time series data into a continuous target memory segment based on a circular queue when the current data does not have periodicity, and to overwrite the historical data in the circular queue with the new data generated by the time series.

[0103] Furthermore, the second processing module includes:

[0104] The current confidence value determination unit is used to determine the current confidence value based on a preset formula and the number of repetitions;

[0105] A confidence value determination and comparison unit is used to compare the magnitude relationship of the confidence values ​​with a preset confidence threshold, and adjust the memory length according to the magnitude relationship.

[0106] Furthermore, the confidence value determination comparison unit includes:

[0107] The memory length adjustment subunit is used to compare the number of data items with the target period length, and adjust the memory length according to the larger value between the number of data items and the target period length.

[0108] Furthermore, the confidence value determination comparison unit includes:

[0109] The first calculation subunit is used to calculate the number of stored data according to a preset formula when the current data does not have periodicity.

[0110] The second calculation subunit is used to ensure that the number of data items is equal to the period length when the current data has periodicity.

[0111] Furthermore, the second processing module also includes:

[0112] The data storage count calculation unit is used to determine the number of data to be stored based on a preset formula when processing data from at least one time point prior to the current time.

[0113] The data writing unit is used to write the target memory data into memory based on the number of stored data and the preset continuous length.

[0114] Furthermore, the period length determination module includes:

[0115] The time window acquisition unit is used to acquire the target time window and the N time windows before the target time when it is determined that the current hash value is equal to the historical hash value;

[0116] The period length determination unit is used to determine the period length of the current data based on the target time window and the N time windows preceding the target time.

[0117] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and modules can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0118] The aforementioned device can be implemented as a computer program, which can be used in, for example... Figure 5 It runs on the device shown.

[0119] Please see Figure 5 Figure 5 This is a schematic block diagram illustrating the structure of a device according to an embodiment of this application. The device may be a server.

[0120] See Figure 5 The device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include non-volatile storage media and internal memory.

[0121] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any time-series-based data processing method.

[0122] The processor provides computing and control capabilities to support the operation of the entire device.

[0123] Internal memory provides an environment for the execution of computer programs stored in non-volatile storage media. When these computer programs are executed by a processor, the processor can perform any time-series-based data processing method.

[0124] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that... Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the device to which the present application is applied. Specific devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0125] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0126] In one embodiment, the processor is configured to run a computer program stored in memory to perform the following steps:

[0127] Obtain the current data at the target time and obtain the hash value of the current data;

[0128] Based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time, the period length L of the current data is determined;

[0129] When the current data does not have a periodicity, the period length is infinite;

[0130] Based on the data length of the corresponding batch processing within the target time window, the data processing delay, and the data generation interval, the memory size is adjusted to achieve processing based on time series data;

[0131] When the current data has a periodicity, the memory size is adjusted based on the period length to achieve processing based on time series data.

[0132] In one embodiment, when the current data does not have periodicity, the period length is infinite. Based on the batch processing data length corresponding to the target time window, the memory size is adjusted to achieve time-series data processing, which is then used to implement:

[0133] When the current data does not have periodicity, the time series data is written to a continuous target memory segment based on a circular queue, and the new data generated by the time series overwrites the historical data in the circular queue.

[0134] In one embodiment, when the current data is periodic, the memory size is adjusted based on the period length to enable time-series data processing, thereby achieving:

[0135] The current confidence value is determined based on a preset formula and the number of repetitions.

[0136] The confidence values ​​are compared with the preset confidence threshold, and the memory length is adjusted according to the comparison.

[0137] In one embodiment, the relationship between the confidence values ​​and a preset confidence threshold is compared, and the memory length is adjusted according to the relationship, to achieve the following:

[0138] The number of data points is compared with the target period length, and the memory length is adjusted based on the larger of the number of data points and the target period length.

[0139] In one embodiment, before comparing the number of data items with the target period length and adjusting the memory length based on the larger of the number of data items and the target period length, the following is implemented:

[0140] When the current data does not exhibit periodicity, the number of stored data items is calculated according to a preset formula, which is as follows:

[0141]

[0142] Where m is the number of stored data, c is the data processing and calculation delay, n is the number of data to be processed in the batch corresponding to the target time window, w is the data generation interval, and math.cealing is integer rounding up;

[0143] When the current data has a periodicity, the number of data points is equal to the period length.

[0144] In one embodiment, when the current data is periodic, the memory size is adjusted based on the period length to enable time-series data processing, thereby achieving:

[0145] When processing data from at least one time point prior to the current time, the number of data items to be stored is determined based on a preset formula;

[0146] Based on the number of stored data items and the preset continuous length, the target memory data is written into memory.

[0147] In one embodiment, the period length of the current data is determined based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time, for the purpose of:

[0148] When it is determined that the current hash value is equal to the historical hash value, the target time window and the N time windows before the target time are obtained;

[0149] The period length of the current data is determined based on the target time window and the N time windows preceding the target time.

[0150] The embodiments of this application also provide a computer-readable storage medium storing a computer program, the computer program including program instructions, and the processor executing the program instructions to implement any of the time-series-based data processing methods provided in the embodiments of this application.

[0151] The computer-readable storage medium can be an internal storage unit of the device described in the foregoing embodiments, such as the hard disk or memory of the device. The computer-readable storage medium can also be an external storage device of the device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the device.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A time series-based data processing method, characterized by, The time-series-based data processing method includes: Obtain the current data at the target time and obtain the hash value of the current data; Based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time, the period length L of the current data is determined; When the current data does not have a periodicity, the period length is infinite; Based on the data length, data processing delay, and data generation interval of the corresponding batch processing within the target time window, the memory size is adjusted to achieve processing based on time series data; When the current data exhibits periodicity, the memory size is adjusted based on the period length to enable time-series data processing, including: The current confidence value is determined based on a preset formula and the number of repetitions. The comparison between the current confidence value and a preset confidence threshold, and the adjustment of the memory size based on the comparison, includes: comparing the number of data points with the target period length, and adjusting the memory size based on the larger of the number of data points and the target period length; Before comparing the number of data points with the target period length, and adjusting the memory size based on the larger of the number of data points and the target period length, the process includes: When the current data does not exhibit periodicity, the number of stored data items is calculated according to a preset formula, which is as follows: Where m is the number of stored data, c is the data processing and calculation delay, n is the number of data to be processed in the batch corresponding to the target time window, w is the data generation interval, and math.cealing is integer rounding up; When the current data has a periodicity, the number of data points is equal to the period length.

2. The time-series-based data processing method according to claim 1, characterized in that, After adjusting the memory size based on the batch processing data length corresponding to the target time window to achieve time-series-based data processing, the following steps are included: When the current data does not have periodicity, the time series data is written to a continuous target memory segment based on a circular queue, and the new data generated by the time series overwrites the historical data in the circular queue.

3. The time-series-based data processing method according to claim 1, characterized in that, When the current data exhibits periodicity, adjusting the memory size based on the period length to achieve time-series data processing includes: When processing data from at least one time point prior to the current time, the number of data items to be stored is determined based on a preset formula; Based on the number of stored data items and the preset continuous length, the target memory data is written into memory.

4. The time-series-based data processing method according to any one of claims 1 to 3, characterized in that, Determining the period length of the current data based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time includes: When it is determined that the current hash value is equal to the historical hash value, the target time window and the N time windows before the target time are obtained; The period length of the current data is determined based on the target time window and the N time windows preceding the target time.

5. A time-series-based data processing device, characterized in that, The apparatus employing the time-series-based data processing method according to any one of claims 1 to 4 comprises: The hash value acquisition module is used to acquire the current data at the target time and acquire the hash value of the current data; The period length determination module is used to determine the period length L of the current data based on the current hash value of the target time window and the historical hash values ​​of the N time windows preceding the target time. The infinite period module is used when the current data does not have periodicity, and the period length is infinite. The first processing module is used to adjust the memory size based on the data length, data processing delay and data generation interval of the corresponding batch processing of the target time window to realize the processing of time series data; The second processing module is used to adjust the memory size based on the period length when the current data has periodicity in order to achieve processing based on time series data, including: The current confidence value is determined based on a preset formula and the number of repetitions. The comparison between the current confidence value and a preset confidence threshold, and the adjustment of the memory size based on the comparison, includes: comparing the number of data points with the target period length, and adjusting the memory size based on the larger of the number of data points and the target period length; Before comparing the number of data points with the target period length, and adjusting the memory size based on the larger of the number of data points and the target period length, the process includes: When the current data does not exhibit periodicity, the number of stored data items is calculated according to a preset formula, which is as follows: Where m is the number of stored data, c is the data processing and calculation delay, n is the number of data to be processed in the batch corresponding to the target time window, w is the data generation interval, and math.cealing is integer rounding up; When the current data has a periodicity, the number of data points is equal to the period length.

6. A device, characterized in that, The device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and, in executing the computer program, implement the time-series-based data processing method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to implement the time-series-based data processing method as described in any one of claims 1 to 4.

Images