Time sequence data processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a time sequence data processing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining initial time sequence data adopted by preset experimental equipment; preprocessing the initial time sequence data to obtain first time sequence data; and performing data compression on the first time sequence data by adopting a preset data compression mode to obtain second time sequence data, the data compression mode being a method for performing data compression based on timestamp information of the first time sequence data. Through the embodiment of the invention, the storage amount of the time sequence data can be reduced, so that the delay in the data sending process can be reduced, the bandwidth can be reduced, and the requirement of data instantaneity can be met.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a time series data processing method and device, electronic equipment, and storage medium. Background Art

[0002] With the advent of the big data era in China, data visualization has begun to gain attention. However, there are still deficiencies in the processing of laboratory data. When processing large-scale data sets, existing tools may not perform well, resulting in low efficiency or the inability to display complex data relationships.

[0003] At present, there are a large number of experimental devices, and the frequency of laboratory data sent by the experimental devices is high. Therefore, the data that needs to be recorded and stored is large, and there is a large delay when sending data, which occupies a large bandwidth and cannot meet the real-time requirements of data. Summary of the Invention

[0004] In view of the above problems, a time series data processing method and device, electronic device, and storage medium are proposed to overcome or at least partially solve the above problems, including:

[0005] A time series data processing method, the method comprising:

[0006] Obtaining initial time series data used by the preset experimental equipment;

[0007] After preprocessing the initial time series data, first time series data is obtained;

[0008] The first time series data is compressed using a preset data compression method to obtain second time series data. The data compression method is a method for compressing data based on timestamp information of the first time series data.

[0009] Optionally, it also includes:

[0010] According to the acquisition timestamp of the first time series data;

[0011] determining a collection interval of the first time series data based on the collection timestamp;

[0012] When the collection interval is fixed, determining that the preset data compression mode is a first data compression mode;

[0013] When the collection interval is not fixed, the preset data compression mode is determined to be the second data compression mode.

[0014] Optionally, compressing the first time series data using a preset data compression method to obtain second time series data includes:

[0015] When the preset data compression mode is the first data compression mode, the first time series data are sorted according to their acquisition timestamps, and multiple adjacent first time series data with the same experimental values ​​are divided into a time series data group;

[0016] Determining a start acquisition timestamp in the time series data group, and determining quantity information of the first time series data in the time series data group;

[0017] The start acquisition timestamp, the acquisition interval, the experimental value, and the quantity information are combined according to a first preset data structure to generate second time series data.

[0018] Optionally, compressing the first time series data using a preset data compression method to obtain second time series data includes:

[0019] When the preset data compression mode is the second data compression mode, the first time series data is RLE encoded and a timestamp is added to each time series data to obtain second time series data.

[0020] Optionally, compressing the first time series data using a preset data compression method to obtain second time series data includes:

[0021] When the first time series data is continuous data, sorting the data according to the acquisition timestamps of the first time series data, and dividing multiple adjacent first time series data with the same experimental values ​​into a time series data group;

[0022] Determining a start acquisition timestamp and duration in the time series data group;

[0023] Determining quantity information of first time series data in the time series data group;

[0024] The start acquisition timestamp, the duration, the experimental value, and the quantity information are combined according to a preset data structure to generate second time series data.

[0025] Optionally, it also includes:

[0026] Obtaining an error factor of the experimental equipment corresponding to the experimental equipment inspection capability;

[0027] Determining whether two adjacent first time series data are within a preset error range corresponding to the error factor;

[0028] When the two adjacent first time series data are within the preset error range corresponding to the error factor, the two adjacent time series data are recorded as identical data.

[0029] Optionally, the preprocessing of the initial time series data to obtain the first time series data includes:

[0030] Filtering target time series data from the initial time series data according to preset data filtering rules;

[0031] The target time series data is converted according to a preset data format to obtain first time series data.

[0032] Optionally, it also includes:

[0033] Restoring the second time series data to the first time series data;

[0034] A timing diagram is generated based on the first timing data, and the timing diagram is displayed.

[0035] Optionally, it also includes:

[0036] Configuring a standard line of time series data in the time series diagram;

[0037] When the first time series data in the time series diagram is not within the standard line range, a warning prompt message is generated.

[0038] A time series data device, comprising:

[0039] An initial time series data acquisition module is used to acquire the initial time series data used by the preset experimental equipment;

[0040] A first time series data generating module, configured to preprocess the initial time series data to obtain first time series data;

[0041] The second time series data generation module is configured to compress the first time series data using a preset data compression method to obtain second time series data. The data compression method is a method of compressing data based on the timestamp information of the first time series data.

[0042] An electronic device includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the above-mentioned time series data processing method when executed by the processor.

[0043] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the time series data processing method described above is implemented.

[0044] The embodiments of the present invention have the following advantages:

[0045] In an embodiment of the present invention, initial time series data used by a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; and the first time series data is compressed using a preset data compression method based on the timestamp information of the first time series data to obtain second time series data. Data compression reduces the amount of data stored, thereby reducing delays and bandwidth during data transmission, thereby meeting data real-time requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the description of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 This is a flowchart of a time series data processing method provided by one embodiment of the present invention;

[0048] Figure 2 is a flowchart of another time series data processing method provided by one embodiment of the present invention;

[0049] Figure 3 is a flowchart of another time series data processing method provided by one embodiment of the present invention;

[0050] Figure 4 is a flowchart of another time series data processing method provided by one embodiment of the present invention;

[0051] Figure 5 is a flowchart of another time series data processing method provided by one embodiment of the present invention;

[0052] Figure 6 1 is a schematic diagram of the structure of a time series data processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are also within the scope of protection of the present invention.

[0054] Reference Figure 1 , which shows a flowchart of a time series data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0055] Step S101, obtaining initial time series data used by a preset experimental device;

[0056] In practical applications, the experimental equipment can be a device for collecting experimental data. The experimental equipment can collect experimental data in a set environment according to user needs. In an embodiment of the present invention, the experimental equipment can collect initial time series data. The initial time series data is time series data with time characteristics, specifically a series of data points arranged in chronological order, each data point is associated with a specific timestamp. The timestamp associated with the initial time series data can be the time when the experimental equipment collects each experimental data point, that is, the collection timestamp.

[0057] Step S102, preprocessing the initial time series data to obtain first time series data;

[0058] In an embodiment of the present invention, after collecting the initial time series data, the initial time series data can be preprocessed. The purpose of preprocessing is to facilitate subsequent compression processing of the data. After preprocessing, the initial time series data can become the first time series data.

[0059] Preprocessing methods can include data cleaning and data conversion. Data cleaning can specifically include handling missing values, handling outliers, and deduplication. Missing value handling involves filling missing values ​​(e.g., using the mean, median, mode, or a predictive model) or deleting records with missing values. Outlier handling involves detecting and processing outliers, and statistical methods or visualization tools can be used during outlier handling. Data deduplication involves removing duplicate records to ensure data uniqueness.

[0060] In one embodiment of the present invention, the preprocessing of the initial time series data to obtain the first time series data includes: filtering out target time series data from the initial time series data according to a preset data filtering rule; and converting the target time series data according to a preset data format to obtain the first time series data.

[0061] In actual applications, after obtaining the initial time series data, the initial time series data can be filtered. Specifically, data filtering rules can be pre-configured, and then the initial time series data can be filtered according to the data filtering rules. The data filtering rules may include one or more filtering conditions. The filtering conditions can be set according to user needs, and there are no excessive restrictions on this in the embodiments of the present invention.

[0062] In one example, the data screening conditions can be set to be associated with factors such as data format, data integrity, time, and rationality. Specifically, it can include checking whether the format of the initial time series data is correct, whether the initial time series data is missing, whether the time of the initial time series data is correct, whether the initial time series data is reasonable, etc.

[0063] After data screening, the target time series data can be standardized, i.e., converted according to a preset data format. After this standardization, first time series data in the preset data format can be obtained. In the embodiments of the present invention, data standardization can achieve a unified data format, facilitating subsequent data compression processing.

[0064] In step S103 , a preset data compression method is used to compress the first time series data to obtain second time series data. The data compression method is a method of compressing data based on the timestamp information of the first time series data.

[0065] After obtaining the first time series data, a preset data compression processing method can be applied to achieve unified compression of large quantities of data to obtain the second time series data, thereby reducing the amount of data processing. After compression, the data occupies less memory, and thus when sending data, data delay can be reduced and bandwidth pressure can be reduced.

[0066] In an embodiment of the present invention, the data compression method used may be Run-Length Encoding (RLE), which is suitable for sequences with a large number of repeated characters or data blocks. The basic idea of ​​RLE is to use a counter and a character to represent consecutive repeated characters, thereby reducing storage space.

[0067] In an embodiment of the present invention, the second time series data obtained after compression processing can be sent to other modules for processing.

[0068] In one embodiment of the present invention, the second time series data is restored to the first time series data; a time series graph is generated based on the first time series data, and the time series graph is displayed.

[0069] In practical applications, the compressed second time series data can be restored to the first time series data. The second time series data is relatively lightweight and can be used for data transmission, while the first time series data can be used for actual data processing. The first time series data and the second time series data can be quickly converted. Specifically, the first time series data is compressed using a data compression method to obtain the second time series data, and the second time series data is decompressed using a data decompression method corresponding to the data compression method to obtain the first time series data.

[0070] After obtaining the first time series data, a visual time series graph can be generated based on the first time series data. The time series graph facilitates users to quickly view the characteristics of the time series data. After generating the time series graph, the time series graph can be displayed.

[0071] In one embodiment of the present invention, a standard line for time series data is configured in the time series diagram; when the first time series data in the time series diagram is not within the range of the standard line, a warning prompt message is generated.

[0072] The standard line may include an upper limit and / or a lower limit. The standard line may be set according to a critical value of the warning, specifically according to the actual needs of the user, and the embodiment of the present invention does not impose too many restrictions on this.

[0073] In actual applications, when the first time series data is within the standard range, it means that the first time series data is normal data. When the second time series data is not within the preset range, it means that the first time series data is abnormal data. Therefore, an early warning prompt message can be generated according to the preset message format, and the early warning prompt message can be used to remind maintenance personnel to check the fault in the form of interface text, broadcast, text message, etc.

[0074] In an embodiment of the present invention, initial time series data used by a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; and the first time series data is compressed using a preset data compression method based on the timestamp information of the first time series data to obtain second time series data. Data compression reduces the amount of data stored, thereby reducing delays and bandwidth during data transmission, thereby meeting data real-time requirements.

[0075] Reference Figure 2 , which shows a flowchart of another time series data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0076] Step S201, obtaining initial time series data used by a preset experimental device;

[0077] In practical applications, the experimental equipment can be a device for collecting experimental data. The experimental equipment can collect experimental data in a set environment according to user needs. In an embodiment of the present invention, the experimental equipment can collect initial time series data. The initial time series data is time series data with time characteristics, specifically a series of data points arranged in chronological order, each data point is associated with a specific timestamp. The timestamp associated with the initial time series data can be the time when the experimental equipment collects each experimental data point, that is, the collection timestamp.

[0078] Step S202: preprocessing the initial time series data to obtain first time series data;

[0079] In an embodiment of the present invention, after collecting the initial time series data, the initial time series data can be preprocessed. The purpose of preprocessing is to facilitate subsequent compression processing of the data. After preprocessing, the initial time series data can become the first time series data.

[0080] Preprocessing methods can include data cleaning and data conversion. Data cleaning can specifically include handling missing values, handling outliers, and deduplication. Missing value handling involves filling missing values ​​(e.g., using the mean, median, mode, or a predictive model) or deleting records with missing values. Outlier handling involves detecting and processing outliers, and statistical methods or visualization tools can be used during outlier handling. Data deduplication involves removing duplicate records to ensure data uniqueness.

[0081] In one embodiment of the present invention, an error factor of the experimental equipment corresponding to the experimental equipment inspection capability is obtained; it is determined that two adjacent first time series data are within a preset error range corresponding to the error factor; when the two adjacent first time series data are within the preset error range corresponding to the error factor, the two adjacent time series data are recorded as the same data.

[0082] The error factor is the error of the experimental equipment itself. The error factor can be determined based on the factory parameters of the experimental equipment, or can be calculated when calibrating the experimental equipment.

[0083] In practical applications, the detection capabilities of the experimental equipment itself may affect the data it measures. After obtaining the first time series data, the first time series data can be sorted in chronological order. Then, for any adjacent first time series data, the error value can be calculated to determine whether the error value is within the preset range of the experimental equipment itself. When the error is within the preset range, it can be determined that the two have the same experimental value. When the error is not within the preset range, no special processing is required.

[0084] In one embodiment of the present invention, according to the collection timestamp of the first time series data; the collection interval of the first time series data is determined based on the collection timestamp; when the collection interval is fixed, the preset data compression method is determined to be the first data compression method; when the collection interval is not fixed, the preset data compression method is determined to be the second data compression method.

[0085] In practical applications, each first time series data item has a corresponding collection timestamp. Multiple first time series data items can be sorted by collection timestamp, and the collection interval between two adjacent first time series data items can be obtained based on the collection timestamps of the two first time series data items. That is, collection interval = |time series data item n - time series data item (n+1)|.

[0086] Step S203: When the preset data compression mode is the first data compression mode, the first time series data are sorted according to their acquisition timestamps, and multiple adjacent first time series data with the same experimental values ​​are grouped into a time series data group;

[0087] In the embodiment of the present invention, the data compression method may be a Run-Length Encoding (RLE) algorithm, which can achieve data compression by representing repeated data with a smaller amount of data.

[0088] For example, a set of first time series data is sorted in chronological order to obtain: 2223344777, which can be divided into four time series data groups: 222, 33, 44, and 777.

[0089] Step S204, determining the start acquisition timestamp in the time series data group, and determining the quantity information of the first time series data in the time series data group;

[0090] Step S205 : Generate second time series data by combining the start acquisition timestamp, the acquisition interval, the experimental value, and the quantity information according to the first preset data structure.

[0091] The data structure of the second time series data is (starting collection timestamp, collection interval, experimental value, quantity information).

[0092] If the data is collected at fixed time intervals, it is possible to compress the data by recording only the timestamp of the start of the sequence and the interval between data collections. The precise time of subsequent data points can then be inferred from the start time and the interval.

[0093] For example, if the experimental equipment starts sending the first time series data at 12:00, the data is 12.0, and the interval is 10 seconds, then the time series data can be recorded as (12:00, 12, 10, x). Here, x is the number of consecutive transmissions, 12.0. Therefore, repeated data can be compressed into one, and the time of each node can be restored from the data, that is, the time of each node = start time + interval time * number of times.

[0094] In an embodiment of the present invention, initial time series data used by a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; when the preset data compression method is the first data compression method, the first time series data is sorted according to the acquisition timestamps of the first time series data, and multiple adjacent first time series data with the same experimental values ​​are divided into a time series data group; the starting acquisition timestamp in the time series data group is determined, and the quantity information of the first time series data in the time series data group is determined; the starting acquisition timestamp, the acquisition interval, the experimental value, and the quantity information are combined according to the first preset data structure to generate second time series data, and the data compression method is a method of compressing data based on the timestamp information of the first time series data. Therefore, through data compression, the amount of data storage is reduced, and the delay in the data transmission process can be reduced, the bandwidth can be reduced, and the real-time requirements of data can be met.

[0095] Reference Figure 3 , which shows a flowchart of another time series data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0096] Step S301, obtaining initial time series data used by a preset experimental device;

[0097] In practical applications, the experimental equipment can be a device for collecting experimental data. The experimental equipment can collect experimental data in a set environment according to user needs. In an embodiment of the present invention, the experimental equipment can collect initial time series data. The initial time series data is time series data with time characteristics, specifically a series of data points arranged in chronological order, each data point is associated with a specific timestamp. The timestamp associated with the initial time series data can be the time when the experimental equipment collects each experimental data point, that is, the collection timestamp.

[0098] Step S302: preprocessing the initial time series data to obtain first time series data;

[0099] In an embodiment of the present invention, after collecting the initial time series data, the initial time series data can be preprocessed. The purpose of preprocessing is to facilitate subsequent compression processing of the data. After preprocessing, the initial time series data can become the first time series data.

[0100] Preprocessing methods can include data cleaning and data conversion. Data cleaning can specifically include handling missing values, handling outliers, and deduplication. Missing value handling involves filling missing values ​​(e.g., using the mean, median, mode, or a predictive model) or deleting records with missing values. Outlier handling involves detecting and processing outliers, and statistical methods or visualization tools can be used during outlier handling. Data deduplication involves removing duplicate records to ensure data uniqueness.

[0101] In one embodiment of the present invention, according to the collection timestamp of the first time series data; the collection interval of the first time series data is determined based on the collection timestamp; when the collection interval is fixed, the preset data compression method is determined to be the first data compression method; when the collection interval is not fixed, the preset data compression method is determined to be the second data compression method.

[0102] In step S303 , when the preset data compression method is the second data compression method, RLE encoding is performed on the first time series data and a timestamp is added to each time series data to obtain second time series data.

[0103] For example, when the time interval for collecting time series data is not fixed, or there is no need to be able to randomly access the time series data in the time series, after RLE encoding, a timestamp needs to be recorded for each RLE-encoded time series data.

[0104] In an embodiment of the present invention, initial time series data used in a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; and when the preset data compression method is the second data compression method, the first time series data is RLE encoded and a timestamp is added to each time series data to obtain second time series data. The data compression method is a method of compressing data based on the timestamp information of the first time series data. This can reduce the storage capacity of time series data, thereby reducing the delay in data transmission and bandwidth, thereby meeting the real-time requirements of data.

[0105] Reference Figure 4 , which shows a flowchart of another time series data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0106] Step 401: obtaining initial time series data used by a preset experimental device;

[0107] In practical applications, the experimental equipment can be a device for collecting experimental data. The experimental equipment can collect experimental data in a set environment according to user needs. In an embodiment of the present invention, the experimental equipment can collect initial time series data. The initial time series data is time series data with time characteristics, specifically a series of data points arranged in chronological order, each data point is associated with a specific timestamp. The timestamp associated with the initial time series data can be the time when the experimental equipment collects each experimental data point, that is, the collection timestamp.

[0108] Step 402: preprocess the initial time series data to obtain first time series data;

[0109] In an embodiment of the present invention, after collecting the initial time series data, the initial time series data can be preprocessed. The purpose of preprocessing is to facilitate subsequent compression processing of the data. After preprocessing, the initial time series data can become the first time series data.

[0110] Preprocessing methods can include data cleaning and data conversion. Data cleaning can specifically include handling missing values, handling outliers, and deduplication. Missing value handling involves filling missing values ​​(e.g., using the mean, median, mode, or a predictive model) or deleting records with missing values. Outlier handling involves detecting and processing outliers, and statistical methods or visualization tools can be used during outlier handling. Data deduplication involves removing duplicate records to ensure data uniqueness.

[0111] Step 403: When the first time series data is continuous data, the first time series data are sorted according to their acquisition timestamps, and multiple adjacent first time series data with the same experimental values ​​are grouped into a time series data group.

[0112] Step 404, determining the start acquisition timestamp and duration of the time series data group;

[0113] The duration is calculated as the difference between the earliest and latest start acquisition timestamps in the time series data group.

[0114] Step 405: Determine the quantity information of the first time series data in the time series data group;

[0115] Step 406 : Generate second time series data by combining the start acquisition timestamp, duration, experimental value, and quantity information according to a preset data structure.

[0116] For each continuous piece of data, record its starting timestamp and the duration of the data value (which can be seconds, minutes, hours, etc.). In this way, you can know when it starts and when it ends, and the data remains unchanged.

[0117] For example, suppose the experimental equipment collects the following time series data:

[0118] Time: 12:00 12:01 12:02 12:03 12:04 12:05...

[0119] Data: 10.0 10.1 10.0 11.0 11.1 13.0...

[0120] If the error factor is 0.2, when using RLE for compression, the time information is recorded. This time series data can be expressed as:

[0121] (12:00,3,10.0)(12:03,2,11.0)(12:05,1,12.0)...

[0122] The first value of each tuple is the timestamp of the start of the data segment (i.e., the starting timestamp), the second value is the number of consecutive repetitions of the data value (i.e., the quantity information), and the third value is the actual data value (i.e., the experimental value). This clearly indicates that the data value is 10.0 in the three minutes from 12:00 to 12:02, and 11.0 in the two minutes from 12:03 to 12:04.

[0123] In an embodiment of the present invention, initial time series data used by a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; when the first time series data is continuous data, the first time series data is sorted according to its acquisition timestamp, and multiple adjacent first time series data with the same experimental value are divided into a time series data group; the starting acquisition timestamp and duration of the time series data group are determined; the quantity information of the first time series data in the time series data group is determined; and the starting acquisition timestamp, duration, experimental value, and quantity information are combined according to a preset data structure to generate second time series data. Thus, through data compression, the amount of data storage is reduced, which can further reduce the delay in the data transmission process, reduce bandwidth, and meet the requirements of data real-time performance.

[0124] Reference Figure 5 , which shows a flowchart of another time series data processing method provided by an embodiment of the present invention, which may specifically include the following steps:

[0125] Step S501: data collection;

[0126] The experimental equipment first needs to collect raw data.

[0127] Step S502: initial screening;

[0128] After obtaining the initial time series data, the initial time series data can be filtered. Specifically, data filtering rules can be pre-configured, and then the initial time series data can be filtered according to the data filtering rules. The data filtering rules may include one or more filtering conditions. The filtering conditions can be set according to user needs, and there are no excessive restrictions on this in the embodiments of the present invention.

[0129] In one example, the data screening conditions can be set to be associated with factors such as data format, data integrity, time, and rationality. Specifically, it can include checking whether the format of the initial time series data is correct, whether the initial time series data is missing, whether the time of the initial time series data is correct, whether the initial time series data is reasonable, etc.

[0130] Step S503: data formatting;

[0131] The obtained target time series data may be subjected to standardization processing, that is, converted according to a preset data format. After the standardization processing, first time series data in the preset data format may be obtained.

[0132] Step S504: data compression;

[0133] By obtaining the error factor of the test equipment's inspection capability, when making data error judgments, if the previous and subsequent data are within the error factor, they are considered identical. When using Run-Length Encoding (RLE) compression on time series data, in addition to recording the data values ​​and their occurrence counts, the time information of the data also needs to be recorded. This time information is usually given in the form of a timestamp, which marks the start time of the data series or the time of a specific data point. The timestamp can be an absolute time (such as UTC time) or a relative time (such as an offset from the start of the series).

[0134] Specifically, they can include the following three types:

[0135] (1) If data collection is performed at fixed time intervals, only the timestamp of the start of the sequence and the time interval of data collection can be recorded when compressing the data.

[0136] The start acquisition timestamp, acquisition interval, experimental value, and quantity information are combined according to the first preset data structure to generate the second time series data. That is, the data structure of the second time series data is (start acquisition timestamp, acquisition interval, experimental value, quantity information).

[0137] For example, if the experimental equipment starts sending the first time series data at 12:00, the data is 12.0, and the interval is 10 seconds, then the time series data can be recorded as (12:00, 12, 10, x). Here, x is the number of consecutive transmissions, 12.0. Therefore, repeated data can be compressed into one, and the time of each node can be restored from the data, that is, the time of each node = start time + interval time * number of times.

[0138] (2) When the time interval for collecting time series data is not fixed, or when there is no need to be able to randomly access the time series data in the time series, a timestamp needs to be recorded for each RLE-encoded time series data after RLE encoding.

[0139] (3) For each continuous data segment, record its starting timestamp and the duration of the data value (which can be seconds, minutes, hours, etc.)

[0140] For example, suppose the experimental equipment collects the following time series data:

[0141] Time: 12:00 12:01 12:02 12:03 12:04 12:05...

[0142] Data: 10.0 10.1 10.0 11.0 11.1 13.0...

[0143] If the error factor is 0.2, when using RLE for compression, the time information is recorded. This time series data can be expressed as:

[0144] (12:00,3,10.0)(12:03,2,11.0)(12:05,1,12.0)...

[0145] The first value of each tuple is the timestamp of the start of the data segment (i.e., the starting timestamp), the second value is the number of consecutive repetitions of the data value (i.e., the quantity information), and the third value is the actual data value (i.e., the experimental value). This clearly indicates that the data value is 10.0 in the three minutes from 12:00 to 12:02, and 11.0 in the two minutes from 12:03 to 12:04.

[0146] Step S505: data caching;

[0147] Store the data in a relational database (such as MySQL).

[0148] Step S506: data visualization;

[0149] Time series data can be converted into time series graphs for visualization.

[0150] Step S507, data warning: setting the experimental data standard line in the time series diagram, and performing operations such as warning prompts on the experimental collection data.

[0151] In an embodiment of the present invention, the data can be initially screened at the beginning of the experimental equipment data collection process, and then through data compression, it is possible to reduce delays when sending data, save bandwidth, and improve the real-time performance of data processing. In an experimental time series data environment (such as obtaining experimental equipment collection information in real time), this method can provide fast and efficient data processing capabilities, which is particularly suitable for applications with high real-time requirements. Then, through data restoration, the dynamic changes of experimental data can be intuitively felt on the data analysis platform to achieve monitoring functions.

[0152] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0153] Reference Figure 6 , which shows a schematic structural diagram of a time series data processing device provided by an embodiment of the present invention, which may specifically include the following modules:

[0154] The initial time series data acquisition module 601 is used to acquire the initial time series data used by the preset experimental equipment;

[0155] A first time series data generating module 602 is configured to preprocess the initial time series data to obtain first time series data;

[0156] The second time series data generating module 603 is configured to compress the first time series data using a preset data compression method to obtain second time series data. The data compression method is a method of compressing data based on the timestamp information of the first time series data.

[0157] In one embodiment of the present invention, the apparatus further comprises:

[0158] A timestamp collection module, configured to collect timestamps based on the first time series data;

[0159] a collection interval module, configured to determine a collection interval of the first time series data based on the collection timestamp;

[0160] A first data compression mode determining module, configured to determine that the preset data compression mode is a first data compression mode when the collection interval is fixed;

[0161] The second data compression mode determining module is configured to determine that the preset data compression mode is the second data compression mode when the collection interval is not fixed.

[0162] In one embodiment of the present invention, the second time series data generating module 603 may include:

[0163] a first time series data group division submodule, configured to, when the preset data compression mode is the first data compression mode, sort the first time series data according to their acquisition timestamps and divide adjacent first time series data with the same experimental values ​​into a time series data group;

[0164] A quantity information and start time determination submodule, configured to determine a start acquisition timestamp in the time series data group, and to determine quantity information of the first time series data in the time series data group;

[0165] The second time series data determination submodule is configured to generate second time series data by combining the start acquisition timestamp, the acquisition interval, the experimental value, and the quantity information according to a first preset data structure.

[0166] In one embodiment of the present invention, the second time series data generating module 603 may include:

[0167] The second time series data determination submodule is used to, when the preset data compression method is the second data compression method, perform RLE encoding on the first time series data and add a timestamp to each time series data to obtain second time series data.

[0168] In one embodiment of the present invention, the second time series data generating module 603 includes:

[0169] A second time series data group determination submodule is configured to, when the first time series data is continuous data, sort the first time series data according to their acquisition timestamps and group multiple adjacent first time series data with the same experimental values ​​into a time series data group;

[0170] A start time and duration determination submodule, configured to determine a start acquisition timestamp and duration in the time series data group;

[0171] a quantity information determination submodule, configured to determine the quantity information of the first time series data in the time series data group;

[0172] The second time series data determination submodule is configured to generate second time series data by combining the start acquisition timestamp, the duration, the experimental value, and the quantity information according to a preset data structure.

[0173] In one embodiment of the present invention, the apparatus further comprises:

[0174] An error factor acquisition module, used to obtain the error factor of the experimental equipment corresponding to the experimental equipment inspection capability;

[0175] An error judgment module, configured to judge whether two adjacent first time series data are within a preset error range corresponding to the error factor;

[0176] The same data determination module is configured to record the two adjacent first time series data as the same data when the two adjacent first time series data are within a preset error range corresponding to the error factor.

[0177] In one embodiment of the present invention, the first time series data generating module 602 may include:

[0178] A data screening module, configured to screen target time series data from the initial time series data according to preset data screening rules;

[0179] The first time series data determination module is configured to convert the target time series data according to a preset data format to obtain first time series data.

[0180] In one embodiment of the present invention, the apparatus further comprises:

[0181] a data restoration module, configured to restore the second time series data to the first time series data;

[0182] A timing diagram display module is used to generate a timing diagram based on the first timing data and display the timing diagram.

[0183] In one embodiment of the present invention, the apparatus further comprises:

[0184] A standard line configuration module, configured to configure a standard line of time series data in the time series diagram;

[0185] The warning prompt message generating module is used to generate a warning prompt message when the first time series data in the time series diagram is not within the standard line range.

[0186] In an embodiment of the present invention, initial time series data used by a preset experimental device is obtained; the initial time series data is preprocessed to obtain first time series data; and the first time series data is compressed using a preset data compression method based on the timestamp information of the first time series data to obtain second time series data. Data compression reduces the amount of data stored, thereby reducing delays and bandwidth during data transmission, thereby meeting data real-time requirements.

[0187] An embodiment of the present invention further provides an electronic device, which may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the following time series data processing method is implemented:

[0188] Obtaining initial time series data used by the preset experimental equipment;

[0189] After preprocessing the initial time series data, first time series data is obtained;

[0190] The first time series data is compressed using a preset data compression method to obtain second time series data. The data compression method is a method for compressing data based on timestamp information of the first time series data.

[0191] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the following time series data processing method is implemented:

[0192] Obtaining initial time series data used by the preset experimental equipment;

[0193] After preprocessing the initial time series data, first time series data is obtained;

[0194] The first time series data is compressed using a preset data compression method to obtain second time series data. The data compression method is a method for compressing data based on timestamp information of the first time series data.

[0195] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0196] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0197] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0199] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0200] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0202] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0203] The above is a detailed introduction to a time series data processing method and device, electronic device, and storage medium. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A time series data processing method, characterized in that: The method comprises: Obtaining initial time series data used by a preset experimental device; After preprocessing the initial time series data, first time series data is obtained; The first time series data is compressed using a preset data compression method to obtain second time series data, wherein the data compression method is a method for compressing data based on timestamp information of the first time series data.

2. The method according to claim 1, characterized in that Also includes: According to the acquisition timestamp of the first time series data; Determine a collection interval of the first time series data based on the collection timestamp; When the collection interval is fixed, determining that the preset data compression mode is a first data compression mode; When the collection interval is not fixed, the preset data compression mode is determined to be the second data compression mode.

3. The method according to claim 2, characterized in that The using a preset data compression method to compress the first time series data to obtain second time series data includes: When the preset data compression mode is the first data compression mode, the first time series data are sorted according to the acquisition timestamps of the first time series data, and a plurality of adjacent first time series data with the same experimental values ​​are divided into a time series data group; Determine a start acquisition timestamp in the time series data group, and determine quantity information of the first time series data in the time series data group; The starting acquisition timestamp, the acquisition interval, the experimental value and the quantity information are combined according to a first preset data structure to generate second time series data.

4. The method according to claim 2, characterized in that: The using a preset data compression method to compress the first time series data to obtain second time series data includes: When the preset data compression mode is the second data compression mode, the first time series data is RLE encoded and a timestamp is added to each time series data to obtain second time series data.

5. The method according to claim 2, characterized in that: The using a preset data compression method to compress the first time series data to obtain second time series data includes: When the first time series data is continuous data, the first time series data are sorted according to their acquisition timestamps, and multiple adjacent first time series data with the same experimental values ​​are divided into a time series data group; Determining a start acquisition timestamp and duration in the time series data group; Determine quantity information of first time series data in the time series data group; The start acquisition timestamp, the duration, the experimental value and the quantity information are combined according to a preset data structure to generate second time series data.

6. The method according to claim 2, characterized in that Also includes: Obtaining an error factor of the experimental equipment corresponding to the experimental equipment inspection capability; Determine whether two adjacent first time series data are within a preset error range corresponding to the error factor; When the two adjacent first time series data are within the preset error range corresponding to the error factor, the two adjacent time series data are recorded as the same data.

7. The method according to claim 1, characterized in that The step of preprocessing the initial time series data to obtain first time series data includes: Filtering target time series data from the initial time series data according to preset data filtering rules; The target time series data is converted according to a preset data format to obtain first time series data.

8. The method according to claim 1, characterized in that Also includes: Restoring the second time series data to the first time series data; A timing diagram is generated based on the first timing data, and the timing diagram is displayed.

9. The method according to claim 8, characterized in that Also includes: Configuring a standard line of time series data in the time series diagram; When the first time series data in the time series diagram is not within the standard line range, a warning prompt message is generated.

10. A time series data device, characterized in that: The device comprises: An initial time series data acquisition module is used to acquire the initial time series data used by the preset experimental equipment; A first time series data generating module, configured to obtain first time series data after preprocessing the initial time series data; The second time series data generating module is used to compress the first time series data by using a preset data compression method to obtain second time series data. The data compression method is a method for compressing data based on the timestamp information of the first time series data.

11. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the method for processing time series data as claimed in any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing time series data as claimed in any one of claims 1 to 9 is implemented.

Images