Time sequence data processing method and device, electronic equipment and storage medium

Abstract

The invention discloses a time sequence data processing method and device, electronic equipment and a storage medium, and relates to the technical field of the Internet of Things, and the method comprises the steps: based on the data field content of each piece of time sequence data, configuring a respective class code, obtaining a timestamp deviation value between the class code and the previous time sequence data, and storing the timestamp deviation value; integrating and generating timestamp processing results corresponding to the time sequence data; obtaining a category code difference group and a timestamp difference group based on the timestamp processing result of each time sequence data; respectively counting different differential values and continuous occurrence times in the category code differential group and the timestamp differential group, and integrating to obtain differential group information; and integrating, obtaining and storing the time sequence data processed information. The time sequence data is classified, the time sequence is recorded according to the timestamp deviation value, a large amount of time sequence data is compressed and stored, the time sequence and uniformity of data storage can be guaranteed, down-conversion sampling of the data is supported, and the time sequence accuracy is guaranteed.

Description

Technical Field

[0001] This application relates to the field of Internet of Things (IoT) technology, specifically to a time-series data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, in the Internet of Things (IoT) field, such as connected vehicles, industrial IoT, smart sensors, drones, and robots, IoT terminals like smart cars, industrial robots, smart sensors, drones, and androids generate massive amounts of time-series data during operation. This data is crucial for the functional interaction and operational status monitoring of IoT terminals. Specific data processing methods need to be selected based on the characteristics of different application scenarios before further application. However, the volume of this data from IoT terminals is extremely large. Common data storage and processing methods such as BLF, MDF, ASC, and Parquet result in very large data volumes and unsatisfactory compression effects, leading to significant storage and transmission costs, as well as low query efficiency.

[0003] To meet practical needs, a time-series data processing technique is proposed. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this application aims to provide a time-series data processing method, apparatus, electronic device, and storage medium that overcomes the deficiencies of the prior art. It classifies time-series data by configuring corresponding category codes based on the data field content, and then records the time sequence using timestamp deviation. By compressing and storing large amounts of time-series data using the aforementioned key information, it ensures the temporal order and uniformity of data storage, supports rapid down-sampling of data while maintaining very high time-series accuracy, and results in smaller, more space-efficient data after down-sampling.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows: In a first aspect, this application provides a time-series data processing method, the method comprising the following steps: Retrieve multiple time-series data from the same data source in chronological order; Based on the data field content of each time series data, configure their respective category codes, obtain the timestamp deviation between each and its previous time series data, and integrate to generate the timestamp processing result corresponding to each time series data; Based on the timestamp processing results of each of the time series data, the corresponding category code difference array and timestamp difference array are obtained; The different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array are counted respectively to obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and then integrated to obtain the difference array information; The differential array information, the timestamp information of the first time-series data, and the data field content corresponding to each category code are integrated to obtain and store the processed time-series data information; wherein... The same data field content corresponds to the same category code.

[0006] Based on the above technical solution, according to the data field content of each time series data, its corresponding category code is configured, and the timestamp deviation between each data and its previous time series data is obtained. The results are then integrated to generate timestamp processing results corresponding to each time series data, including the following steps: For each of the aforementioned time-series data, obtain the corresponding category code; Each of the time series data is compared with the time series data preceding it by timestamp to obtain the timestamp deviation between each of the time series data and the time series data preceding it; The category code and timestamp deviation of each of the aforementioned time series data are integrated into the corresponding timestamp processing result.

[0007] Based on the above technical solution, and based on the timestamp processing results of each of the time-series data, the corresponding category code difference array and timestamp difference array are obtained, including the following steps: Extract the category code from the timestamp processing result of each time series data, compare it with the category code of the previous time series data, obtain the category code difference value corresponding to each time series data, and integrate them to obtain the category code difference array; Extract the timestamp deviation from the timestamp processing results of each of the time series data, and integrate them to obtain the timestamp difference array.

[0008] Based on the above technical solution, the different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array are statistically analyzed to obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and then integrated to obtain the difference array information, including the following steps: The different difference values ​​and their corresponding consecutive occurrences in the category code difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding category code difference value-occurrence association data. The category code difference value-occurrence association data are then integrated to obtain the category code difference merged encoding data. The different difference values ​​and their corresponding consecutive occurrences in the timestamp difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding timestamp difference value-count association data. The timestamp difference value-count association data are then integrated to obtain the timestamp difference merged encoding data. Differential array information is obtained by integrating differentially merged encoding data based on category code and differentially merged encoding data based on timestamp.

[0009] Based on the above technical solution, the method further includes the following steps: Extract the differential array information, the timestamp information of the first time series data, and the data field content corresponding to each category code from the processed information of the time series data, and decode to obtain multiple time series data from the same data source.

[0010] Based on the above technical solution, the method further includes a frequency reduction sampling process, which includes the following steps: Based on the differential array information in the processed time-series data, the corresponding category code differential merged encoding data and timestamp differential merged encoding data are obtained, and the corresponding category code differential data group and timestamp differential data group are decoded to generate the corresponding category code differential data group. According to the preset down-rate, the category code differential data group and the timestamp differential data group are sampled to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. For the first data in the category code sampling differential data group and the timestamp sampling differential data group, locate position m in the category code differential data group and the timestamp differential data group, and add the difference value of each unsampled data before the m-th data in the category code differential data group and the timestamp differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; Each data point from the second point onwards in the category code sampling differential data group and the timestamp sampling differential data group is referred to as the i-th data point. The positions m and n of the (i-1)-th and i-th data points in the category code sampling differential data group and the timestamp sampling differential data group are located in the category code differential data group and the timestamp differential data group, respectively. The difference value of the i-th data point in the category code sampling differential data group and the timestamp sampling differential data group is added to the difference value of each unsampled data point between the m-th and n-th data points in the category code differential data group and the timestamp differential data group. Repeat the above process to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group, and obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. Based on the updated category code sampling differential data group and the updated timestamp sampling differential data group, combined with each category code and the corresponding data field content in the time-series data processing information, multiple corresponding down-frequency sampling time-series data are decoded to obtain; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

[0011] Secondly, this application provides a time-series data processing apparatus, the apparatus comprising: The data acquisition module is used to acquire multiple time-series data from the same data source in chronological order. The data comparison module is used to configure the corresponding category code based on the data field content of each time series data, obtain the timestamp deviation between each data and its previous time series data, and integrate them to generate the timestamp processing result corresponding to each time series data. The difference array generation module is used to obtain the corresponding category code difference array and timestamp difference array based on the timestamp processing results of each of the time series data. The difference array sorting module is used to separately count the different difference values ​​and corresponding consecutive occurrences in the category code difference array and the timestamp difference array, obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and integrate them to obtain the difference array information; The data processing module is used to integrate the difference array information, the timestamp information of the first time series data, and the data field content corresponding to each category code to obtain and store the processed time series data information; wherein, The same data field content corresponds to the same category code.

[0012] Based on the above technical solution, the data comparison module is also used to obtain the corresponding category code for the data field content of each time series data; The data comparison module is also used to compare the timestamps of each of the time series data with the timestamps of the previous time series data to obtain the timestamp deviation between each of the time series data and the timestamps of the previous time series data. The data comparison module is also used to integrate the category code and timestamp deviation of each time series data into the corresponding timestamp processing result.

[0013] Based on the above technical solution, the difference array generation module is further used to extract the category code from the timestamp processing result of each time series data, compare it with the category code of the previous time series data, obtain the category code difference value corresponding to each time series data, and integrate them to obtain the category code difference array; The difference array generation module is also used to extract the timestamp deviation from the timestamp processing results of each of the time series data, and integrate them to obtain the timestamp difference array.

[0014] Based on the above technical solution, the differential array sorting module is also used to count the different differential values ​​and their corresponding consecutive occurrences in the category code differential array, associate each differential value with its corresponding consecutive occurrences to generate corresponding category code differential value-occurrence association data, and integrate each category code differential value-occurrence association data to obtain the category code differential merged encoding data. The difference array sorting module is also used to count the different difference values ​​and their corresponding consecutive occurrences in the timestamp difference array, associate each difference value with its corresponding consecutive occurrences to generate corresponding timestamp difference value-count association data, and integrate each timestamp difference value-count association data to obtain the timestamp difference merged encoding data. The differential array sorting module is also used to integrate differential array information based on the class code differential merged encoding data and the timestamp differential merged encoding data.

[0015] Based on the above technical solution, the device further includes: The data decoding module is used to extract the differential array information, the timestamp information of the first time series data, and the data field content corresponding to each category code from the information after processing the time series data, and decode to obtain multiple time series data from the same data source.

[0016] Based on the above technical solution, the device further includes: The data extraction module is used to obtain the corresponding category code differential merged encoding data and timestamp differential merged encoding data based on the differential array information in the time series data processed information, and decode to generate the corresponding category code differential data group and timestamp differential data group. The down-frequency processing module is used to sample the category code differential data group and the timestamp differential data group according to a preset down-frequency ratio to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. The data group processing module is used to locate position m in the first data in the category code sampling differential data group and the timestamp sampling differential data group, and add the difference value of each unsampled data before the m-th data in the category code sampling differential data group and the timestamp sampling differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; The data group processing module is further configured to, for each data item from the second to the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group, locate the (i-1)-th data item and the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group at their corresponding positions m and n in the category code differential data group and the timestamp differential data group, and add the difference value of the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group to the difference value of each unsampled data item between the m-th and n-th data items in the category code differential data group and the timestamp differential data group; The data group processing module is also used to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group to obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. The down-frequency acquisition module is used to decode and obtain multiple corresponding down-frequency sampled time series data based on the updated category code sampled differential data group and the updated timestamp sampled differential data group, combined with each category code and the corresponding data field content in the time series data processed information; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

[0017] Thirdly, this application provides an electronic device, the electronic device comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the timing data processing method as mentioned in the first aspect.

[0018] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the timing data processing method mentioned in the first aspect.

[0019] Compared with the prior art, the advantages of this application are: This application overcomes the shortcomings of the prior art by configuring corresponding category codes according to the data field content of time series data, thereby classifying the time series data, and then recording the time series order by timestamp deviation. By compressing and storing a large amount of time series data through the above key information, the time order and uniformity of data storage can be guaranteed. It supports fast down-sampling of data and ensures very high time series accuracy. The down-sampling data is smaller in size and saves more space. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the steps of a timing data processing method according to an embodiment of this application. Figure 2 This is a structural block diagram of a timing data processing apparatus according to an embodiment of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0024] This application provides a time-series data processing method, apparatus, electronic device, and storage medium, overcoming the shortcomings of the prior art. It configures corresponding category codes according to the data field content of the time-series data to classify the time-series data, and then records the time sequence using timestamp deviation. By compressing and storing a large amount of time-series data through the above key information, it can ensure the time sequence and uniformity of data storage, support fast down-sampling of data and ensure very high time sequence accuracy, and the down-sampling data is smaller and saves more space.

[0025] To achieve the aforementioned technical effects, the overall concept of this application is as follows: A time-series data processing method, comprising the following steps: S1. Obtain multiple time-series data from the same data source in chronological order; S2. Based on the data field content of each time series data, configure the corresponding category code for each data, obtain the timestamp deviation between each data and its previous time series data, and integrate to generate the timestamp processing result corresponding to each time series data. S3. Based on the timestamp processing results of each time series data, obtain the corresponding category code difference array and timestamp difference array; S4. Calculate the different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array respectively, obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and integrate them to obtain the difference array information. S5. Integrate the difference array information, the timestamp information of the first time series data, and the data field content corresponding to each category code to obtain and store the processed time series data information; among which... The same data field content corresponds to the same category code.

[0026] The embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0027] Firstly, see [the following] Figure 1 As shown in the figure, this application provides a time-series data processing method, which includes the following steps: S1. Obtain multiple time-series data from the same data source in chronological order; S2. Based on the data field content of each time series data, configure the corresponding category code for each data, obtain the timestamp deviation between each data and its previous time series data, and integrate to generate the timestamp processing result corresponding to each time series data. S3. Based on the timestamp processing results of each time series data, obtain the corresponding category code difference array and timestamp difference array; S4. Calculate the different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array respectively, obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and integrate them to obtain the difference array information. S5. Integrate the difference array information, the timestamp information of the first time series data, and the data field content corresponding to each category code to obtain and store the processed time series data information; among which... The same data field content corresponds to the same category code.

[0028] It should be noted that the massive amounts of time-series data generated by IoT terminals during operation require a highly efficient and compressed data processing and storage method to process and store this data. Commonly used processing methods in the industry for time-series data, such as BLF, MDF, ASC, and Parquet, generally have limited compression effects. Due to the large amount of data, they bring high storage costs. At the same time, due to the limited storage space of IoT terminals, important data can be quickly overwritten and lost. This application provides a novel technical solution that separates and stores the data field content and timestamp of a data frame. The data field content is clustered, merged, and classified. The timestamp and the corresponding category code are then sequentially differentially merged and mapped to generate a data format with good compression and support for downsampling. This data processing method achieves a very high compression rate, resulting in a very small data size and significantly reducing data storage costs. It allows for the storage of data for a longer period within the limited storage space of IoT terminals. Furthermore, it ensures the temporal order and uniformity of data storage, supports rapid downsampling of data while maintaining high temporal accuracy. The downsampled data is smaller and more space-efficient, simultaneously satisfying the storage requirements of both original precision data and downsampled data. This allows for better integration of the two types of data according to actual needs. For example, original precision data can be retained for short-term storage, while downsampled data can be stored for long-term storage, enabling the storage of data for a longer period within a limited space.

[0029] In this embodiment, the shortcomings of the prior art are overcome. The corresponding category code is configured according to the data field content of the time series data to classify the time series data. Then, the time series order is recorded by the timestamp deviation. By compressing and storing a large amount of time series data through the above key information, the time order and uniformity of data storage can be guaranteed. It supports fast down-sampling of data and ensures very high time series accuracy. The down-sampling data is smaller in size and saves more space.

[0030] Furthermore, based on the data field content of each time series data, a corresponding category code is configured for each, and the timestamp deviation between each and its previous time series data is obtained. These are then integrated to generate a timestamp processing result corresponding to each time series data, including the following steps: For each of the aforementioned time-series data, obtain the corresponding category code; Each of the time series data is compared with the time series data preceding it by timestamp to obtain the timestamp deviation between each of the time series data and the time series data preceding it; The category code and timestamp deviation of each of the aforementioned time series data are integrated into the corresponding timestamp processing result.

[0031] Furthermore, based on the timestamp processing results of each of the aforementioned time-series data, the corresponding category code difference array and timestamp difference array are obtained, including the following steps: Extract the category code from the timestamp processing result of each time series data, compare it with the category code of the previous time series data, obtain the category code difference value corresponding to each time series data, and integrate them to obtain the category code difference array; Extract the timestamp deviation from the timestamp processing results of each of the time series data, and integrate them to obtain the timestamp difference array.

[0032] Furthermore, the different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array are statistically analyzed to obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and then integrated to obtain the difference array information, including the following steps: The different difference values ​​and their corresponding consecutive occurrences in the category code difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding category code difference value-occurrence association data. The category code difference value-occurrence association data are then integrated to obtain the category code difference merged encoding data. The different difference values ​​and their corresponding consecutive occurrences in the timestamp difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding timestamp difference value-count association data. The timestamp difference value-count association data are then integrated to obtain the timestamp difference merged encoding data. Differential array information is obtained by integrating differentially merged encoding data based on category code and differentially merged encoding data based on timestamp.

[0033] Furthermore, the method also includes the following steps: Extract the differential array information, the timestamp information of the first time series data, and the data field content corresponding to each category code from the processed information of the time series data, and decode to obtain multiple time series data from the same data source.

[0034] Furthermore, the method also includes a down-frequency sampling process, which includes the following steps: Based on the differential array information in the processed time-series data, the corresponding category code differential merged encoding data and timestamp differential merged encoding data are obtained, and the corresponding category code differential data group and timestamp differential data group are decoded to generate the corresponding category code differential data group. According to the preset down-rate, the category code differential data group and the timestamp differential data group are sampled to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. For the first data in the category code sampling differential data group and the timestamp sampling differential data group, locate position m in the category code differential data group and the timestamp differential data group, and add the difference value of each unsampled data before the m-th data in the category code differential data group and the timestamp differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; Each data point from the second point onwards in the category code sampling differential data group and the timestamp sampling differential data group is referred to as the i-th data point. The positions m and n of the (i-1)-th and i-th data points in the category code sampling differential data group and the timestamp sampling differential data group are located in the category code differential data group and the timestamp differential data group, respectively. The difference value of the i-th data point in the category code sampling differential data group and the timestamp sampling differential data group is added to the difference value of each unsampled data point between the m-th and n-th data points in the category code differential data group and the timestamp differential data group. Repeat the above process to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group, and obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. Based on the updated category code sampling differential data group and the updated timestamp sampling differential data group, combined with each category code and the corresponding data field content in the time-series data processing information, multiple corresponding down-frequency sampling time-series data are decoded to obtain; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

[0035] It is important to emphasize that, considering the large volume, periodicity, repetitiveness, and finiteness of data within IoT terminals, the technical solution of this application separates and processes the data field content and timestamp of the data frame, and designs different processing schemes according to the different characteristics of the data field content and timestamp data, in order to maximize data compression. The data field content is clustered, merged, and classified. The timestamps and the corresponding category codes are differentially merged and encoded according to the mapping order. Finally, the original precision data is downsampled as needed. Due to the periodicity, finiteness, high frequency, and large volume of data within IoT terminals, clustering and merging data field content can reduce a large amount of redundant data storage and improve data compression rate. Differential merging encoding operations are performed on timestamps and corresponding category codes according to the mapping order. Combining the periodicity and finiteness of the data, the corresponding timestamps and category codes also exhibit periodicity and finiteness. This encoding method can greatly reduce the storage of various redundant data and further improve the overall data compression rate. Compared with traditional data storage methods such as ASC, MDF, and BLF, the compression rate can be improved by 5-30 times or more. Finally, because this processing method maintains good temporal order and uniformity of the data, it can support efficient downsampling to further reduce the overall data volume.

[0036] In brief, the technical solution of this application mainly includes four parts: First, for data under the same data ID, the timestamp and data field content are separated and processed separately; Second, the data field portion of each data item is clustered, deduplicated, and categorized and encoded according to its content before storage. Third, perform differential merging encoding operations on each data timestamp and the corresponding category code according to the mapping order. First, process the timestamp of each data in an independent array according to the data sequence. Then, process the category code of each data in an independent array according to the data sequence. Ensure that the timestamp array and the category code array are mapped one by one in the same order. Then, perform differential processing on the data in each array, and merge and count the differential data. Finally, encode and store the differential value and the count value according to a specific encoding method. Fourth, the complete timestamp data stored in the encoding is decoded according to the decoding method to obtain independent timestamp difference arrays and class code difference arrays. According to the required downsampling ratio n, one sample is taken from each of the timestamp difference arrays and class code difference arrays every n (every) lines. The difference timestamp or difference class code part in each sampled data is processed as follows: its value is added to the difference value of each unsampled data between the current data and the previous sampled data. The processed data is re-encoded and stored to obtain the down-frequency data.

[0037] The specific implementation process based on the technical solution of the embodiments of this application is as follows: First, the timestamp and data field content of the first piece of data received from the data source are processed separately in different data blocks in memory: 1. Data block A primarily processes the data encoding information after clustering and deduplication of the stored data field content; Data block B1 mainly processes and stores the timestamp differential merged encoded data after processing each data item, while data block B2 mainly processes and stores the category code differential merged encoded data after processing each data item. Data block C1 mainly processes and stores the timestamp data of the intermediate state. The initial value of the timestamp in data block C1 can be set to any valid value. Data block C2 primarily processes and stores intermediate class code data. The initial value of the class code in data block C2 is usually set to 1.

[0038] 2. Generate a corresponding category code for the data field content of the first data entry. This category code serves as the category code for that data entry (the first data entry). After pre-encoding the data field content and the category code, store them in data block A. In this scheme, the generation or setting method of the category code is recorded by the storage order position of the clustered data field content in data block A. For example, if a data field content is stored at position 1 in data block A, its category code is recorded as 1; if a data field content is stored at position n in data block A, its category code is recorded as n.

[0039] 3. Subtract the intermediate timestamp value of the current record in data block C1 from the timestamp value of the first data to obtain the differential timestamp information of the data (the first data). Pre-store the differential timestamp information of the data (the first data) in data block B1.

[0040] 4. Store the complete timestamp data of the first data entry in data block C1 as intermediate timestamp data.

[0041] 5. Subtract the intermediate state category code value of the current record in data block C2 from the category code value of the first data to obtain the differential category code information of the data (the first data). Pre-store the differential category code information of the data (the first data) in data block B2.

[0042] 6. Store the category code data of the first data entry in data block C2 as intermediate category code data.

[0043] Second, for the second piece of data received from the data source, its timestamp and data field content are processed separately: 1. First, process the data field content: Within data block A, query whether the data field content, after clustering, contains this data field entry. (1) If the data field content is already included, the category code corresponding to the data field content is taken as the category code of the data (the second data). The data field content is merged by clustering and will not be processed repeatedly. (2) If the data field content is not included, a new category code is generated for the data field content. The data field content and the corresponding category code are pre-encoded and stored in data block A. At the same time, the newly generated category code is used as the category code of the data (the second data).

[0044] 2. Next, process the timestamp part of the data (the second data): Subtract the intermediate timestamp value of the current record in data block C1 from the timestamp value of the data (the second data) to obtain the differential timestamp information of the data (the second data), and pre-store the differential timestamp information of the data (the second data) in data block B1.

[0045] 3. Store the complete timestamp data of the second data in data block C1 as intermediate timestamp data.

[0046] 4. Next, process the category code part of the data (the second data). Subtract the intermediate category code value of the current record in data block C2 from the category code value of the data (the second data) to obtain the differential category code information of the data (the second data). Pre-store the differential category code information of the data (the second data) in data block B2.

[0047] 5. Store the category code data of the second data in data block C2 as intermediate category code data.

[0048] 6. When processing the timestamp and category code portions of this data (the second data), it is necessary to ensure that the order of the pre-storage location of the differential timestamp information in data block B1 and the order of the pre-storage location of the differential category code in data block B2 are completely consistent, forming a one-to-one mapping relationship.

[0049] 7. For data subsequently obtained from this data source, repeat the above process to process its data field content, timestamp, and category code respectively, and store them in the corresponding data block. At the same time, update the intermediate state timestamp information in data block C1 and the intermediate state category code information in C2.

[0050] Third, after retrieving data from the data source, the data is encoded: 1. Serialize the pre-encoded data of the clustered and deduplicated data field within data block A into a storable data field clustering encoded byte string, and store the byte length information of the aforementioned data field clustering encoded byte string within data block A using a byte string of specified length.

[0051] 2. Merge and encode the pre-stored differential timestamp data and differential category code data in data blocks B1 and B2 respectively. The specific scheme for merging and encoding is as follows: 2.1 First, the data is merged. The merging method is to merge and count the data items with the same difference value in the pre-stored differential data in data block B1 or B2 according to the storage order, and generate temporary <difference value-count value> data pairs. For example, the pre-stored differential timestamp data in data block B1 is <20, 20, 20, 20, 30, 30, 20, 20, 20, 25, 25, 25>. After merging, the data is a series of <difference value-count value> data pairs: <<20-4>, <30-2>, <20-3>, <25, 3>>. 2.2 Then, each generated <difference value-count value> data pair is encoded sequentially, and the specific encoding method is as follows: 2.2.1 First, a difference value bit length flag is set to indicate the length of the bit field occupied by the difference value. Second, a difference value positive / negative flag is set to indicate the positive or negative nature of the difference value. Finally, the difference value is recorded using bits of the length of the bit field occupied by the difference value, and the count value is recorded using a bit field of corresponding length. The length of the difference value bit field and the length of the count value bit field satisfy the following principles and relationships: Considering the large amount of data in IoT terminals, the design of the encoding method needs to take into account the high utilization of byte space and avoid useless waste; Considering the periodicity and finiteness of data within IoT terminals, the lengths of the difference value bit field and the count value bit field within the minimum byte space can be designed and calculated using the following two sets of constraints: 1): (T / c) / (2**L1) < 2**L2-1 <= (T / c) / (2**(L1-1)); 2):min(ceil((x+L1+L2) / 8)); in: T represents the data processing time span, c represents the minimum data period, and T / c = the total number of data entries within the processing time span. Based on the finite nature of the data, the maximum number of data field category codes will not exceed T / c. L1 is the length of the bit field used to store the difference value. 2 to the power of L1 represents the maximum difference value that this bit field can record. (T / c) / (2**L1) represents the minimum number of consecutive occurrences that can occur in T / c category codes when the difference value is at its maximum. 2 to the power of (L1-1) represents the minimum difference value that this bit field can record. (T / c) / (2**(L1-1)) represents the maximum number of consecutive occurrences that can occur in T / c category codes when the difference value is at its minimum. L2 is the length of the bit field storing the count value. 2**L2-1 represents the number of consecutive identical difference values ​​that can be recorded by this bit field length. It is necessary to ensure that (T / c) / (2**L1) < 2**L2-1 <= (T / c) / (2**(L1-1)). x represents the length of the difference value bit length flag and the bit length occupied by the positive and negative flags; ceil((x+L1+L2) / 8) represents the length of bytes required for storage; min(ceil((x+L1+L2) / 8)) represents the minimum reasonable byte length for storage to achieve high utilization of byte space and avoid unnecessary waste. Based on practical data processing experience, assuming T is one minute (60000ms), and c is taken as the minimum period value of 10ms, T / c represents the maximum number of class codes in the data field within this time range, and the calculated result is 6000; when the class code remains unchanged or increases continuously, the minimum difference value of the class code is 1 or 0, that is, 2**L1_min=(1,0), and L1_min=1, L2_max=13 can be calculated; when the class code jumps by a large span, the maximum difference value of the class code is 5999, that is, 2**L1_max‎ = (,5999,), and L1_max=13, L2_min‎ = 1; Based on the above two sets of boundary conditions and the analysis of the values ​​of L1_min, L2_max, L1_max, and L2_min, it can be calculated that 3 bytes are needed for encoding and storage.

[0052] Further considering the impact of x on the encoded bytes, a range optimization can be performed to ensure the reasonableness of the values ​​of L1 and L2. In this scheme, the following settings are made when performing differential merging encoding on the category code: Use 4 bits to store the difference value length flag and use 1 bit to store the difference value positive / negative flag. When the absolute value of the class code difference is in the range [0, 63], the difference value bit length flag is set to 0, and the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference. 6 bits are used to store the absolute value of the class code difference, and 13 bits are used to store the count value of the class code difference. When the absolute value of the class code difference is in the range of [64, 255], the difference value bit length flag is set to 1, and the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference. 8 bits are used to store the absolute value of the class code difference, and 7 bits are used to store the count value of the class code difference. When the absolute value of the class code difference is in the range of [256, 1024], the difference value bit length flag is set to 2, the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference, 10 bits are used to store the absolute value of the class code difference, and 5 bits are used to store the count value of the class code difference. When the absolute value of the class code difference is in the range of [1024, 6000], the difference value bit length flag is set to 3, the difference value positive / negative flag is set to 0 or 1 according to the positive or negative value of the difference, 13 bits are used to store the absolute value of the class code difference, and 3 bits are used to store the count value of the class code difference.

[0053] 2.2.2 Regarding the differential merging encoding calculation method for timestamps, the above formula needs to be modified to the following boundary conditions: 1): T / t_max < 2**L2_min - 1; 2): T / t_min < 2**L2_max – 1; 3): min(ceil((x+ceil(log2(t))+L2) / 8)); Where T is the data processing time span, t_max is the maximum data period, t_min is the minimum data period, L2_min is the minimum length of the bit field storing the count value, and L2_max is the maximum length of the bit field storing the count value. Based on the periodicity of the data, it can be assumed that the period is no more than 1000 milliseconds and no less than 10 ms. T / t_max < 2**L2_min - 1 corresponds to the minimum value of L2 when the maximum difference time interval is taken, where t_max=1000. Assuming T=60000ms, which is one minute, then L2_min=6 can be calculated. T / t_min < 2**L2_max - 1 corresponds to the minimum value of L2 when taking the minimum difference time interval, where t_min=10. Assuming T=60000ms, which is one minute, then L2_max=13 can be calculated. In condition 3, ceil(log2(t)) represents the minimum number of bits required to store the time difference value. Combining the above two sets of boundary conditions and t_max, t_min, L2_max, and L2_min, it can be calculated that 3 bytes are needed for encoding and storage.

[0054] Further considering the impact of x on the encoded bytes, and the commonly used data periods of 10ms, 20ms, 50ms, 100ms, 200ms, 500ms, and 1000ms in actual IoT terminals, as well as the extremely unstable data periods in abnormal scenarios, a range optimization is performed on the reasonableness of the values ​​of t and L2. In this scheme, the following settings are made when performing differential merging encoding on the timestamp: Use 1 bit to store the data period abnormality flag, use 3 bits to store the difference value bit length flag, and use 1 bit to store the difference value positive or negative flag. When the absolute value of the timestamp difference is in the range of [0, 63], the period abnormality flag is set to 0, the difference value bit length flag is set to 0, and the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference. 6 bits are used to store the absolute value of the timestamp difference, and 13 bits are used to store the count value of the difference of the category code. When the absolute value of the timestamp difference is in the range of [64, 511], the period abnormality flag is set to 0, the difference value bit length flag is set to 1, and the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference. 9 bits are used to store the absolute value of the timestamp difference, and 10 bits are used to store the count value of the difference of the category code. When the absolute value of the timestamp difference is in the range of [512, 8191], the period abnormality flag is set to 0, the difference value bit length flag is set to 2, the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference, 13 bits are used to store the absolute value of the category code difference, and 6 bits are used to store the count value of the category code difference. When the absolute value of the timestamp difference is in the range of [8192, 60000], the periodic anomaly flag is set to 1, the difference value bit length flag is set to 3, and the difference value positive or negative flag is set to 0 or 1 according to the positive or negative value of the difference. 16 bits are used to store the absolute value of the category code difference, and 3 bits are used to store the count value of the category code difference.

[0055] 2.3. Concatenate the byte strings of each <difference value-count value> data pair after encoding according to the above encoding method in order to obtain the differential merge encoded data byte string after serialization of the entire data block (B1 or B2).

[0056] 2.4 The differential timestamp data pre-stored in data block B1 is processed in the above manner to obtain the timestamp differential merged encoded data; the byte length information of the aforementioned timestamp differential merged encoded data byte string is stored using a byte string of specified length.

[0057] 2.5 The differential category code data pre-stored in data block B2 is processed in the above manner to obtain the differential merged category code encoded data; the byte length information of the aforementioned differential merged category code encoded data byte string is stored using a byte string of specified length.

[0058] The byte string containing the data field clustering encoding byte string length information in data block A, the data field clustering encoding byte string in data block A, the byte string containing the timestamp differential merging encoding byte string length information in data block B1, the timestamp differential merging encoding data in data block B1, the byte string containing the category code differential merging encoding byte string length information in data block B2, and the category code differential merging encoding data in data block B2 are concatenated and stored in a disk file to obtain the original frequency precision storage data.

[0059] Fourth, read the original frequency precision data stored on the disk as described above: 1. Read the data sequentially according to the byte length information, and read the timestamp differential merged encoded byte string data in data block B1 and the category code differential merged encoded byte string data in data block B2; perform the following down-frequency sampling processing on the timestamp differential merged encoded data and the category code differential merged encoded data respectively: The specific process of down-sampling is as follows: 2.1 The differential merged encoded data byte string is split according to the length of each encoded data stored therein (defined as 3 bytes in this scheme through optimized design) to obtain the encoded data of each pair of <differential value-count value> data pairs. Then, the inverse decoding operation is performed on each pair of data pairs to obtain each pair of <differential value-count value> data pairs. Inverse decoding is the reverse operation of encoding. Here, we will only briefly introduce the process without going into detail: Read the 4-bit difference value bit length flag and the 1-bit difference value positive / negative flag in sequence. Use the difference value bit length flag to confirm the bit field length information storing the absolute value of the difference value and the bit field length information storing the difference value count value. Then, according to the corresponding bit field length information, read the absolute value of the difference value and the difference value count value of the corresponding bit length respectively. Confirm the positive / negative of the difference value based on the difference value positive / negative flag. After obtaining each pair of <difference value-count value> data pairs, the difference values ​​are restored to the original difference data group based on the count value, denoted as difference data group A. In the process of restoring the difference data group, it is necessary to ensure that each pair of <difference value-count value> data pairs is restored in the original order to ensure the order of the data.

[0060] 2.2 Sampling differential data group B is obtained by sampling one data point for every n data points in the above differential data group A according to the required down-frequency ratio n.

[0061] 2.3. Perform the following processing on the sampled difference data group B: add the difference value of each data point in it to the difference value of each unsampled data point in data group A between that data point and its preceding sampled data point. The specific operation is as follows: (1) For the first data in data group B, locate the position m of the data in data group A, and add the difference value of the first data in data group B to the difference value of each unsampled data before the mth data in data group A; (2) For each data point from the second point onwards in data group B, which is called the i-th data point, locate the positions m and n of the i-1 data point and the i-th data point in data group A, and add the difference value of the i-th data point in data group B to the difference value of each unsampled data point between the m-th and n-th data points in data group A. (3) Repeat this process to update the difference value of each data in data group B.

[0062] 2.4. Encode the data in data group B using the differential merging encoding operation described in step 3 above, thus completing the down-sampling processing of the data.

[0063] 3. Perform the above-described down-sampling process on the timestamp differential merged encoded data to obtain down-frequency timestamp differential merged encoded data; use a specified length byte string to store the byte length information of the aforementioned down-frequency timestamp differential merged encoded data byte string.

[0064] 4. Perform the above-described down-sampling process on the differential merged category code data to obtain the down-sampled differential merged category code data; store the byte length information of the aforementioned down-sampled differential merged category code data byte string using a specified length byte string.

[0065] 5. Replace the byte string containing the byte length information of the timestamp differential merging encoded data, the timestamp differential merging encoded data, the byte string containing the byte length information of the category code differential merging encoded data, and the category code differential merging encoded data in the original frequency precision data with the byte string containing the timestamp differential merging encoded data in data block B1, the timestamp differential merging encoded data in data block B1, the byte string containing the byte length information of the category code differential merging encoded data in data block B2, and the category code differential merging encoded data in data block B2. The data obtained after replacement is the data obtained by down-frequency reduction of the original precision data.

[0066] Based on the technical solution of the embodiments of this application, a specific implementation method is given as follows: The raw data consists of: timestamp + data field content, as shown in the example below: 0;oxff,oxfe,oxfd,oxfd; 10;oxff,oxfe,oxfd,oxfd; 20;oxff,oxff,oxfd,oxfd; 30;oxff,oxff,oxfd,oxfd; 41;oxff,oxff,oxfd,oxfd; 51;oxff,oxfe,oxfe,oxfd; 61;oxff,oxfe,oxfe,oxfd; 70;oxff,oxfe,oxfd,oxfd; 80;oxff,oxfe,oxfd,oxfd; 90;oxff,oxfe,oxfd,oxfd; 100;oxff,oxfe,oxfd,oxfd.

[0067] Step 1, Data Field Clustering: 1.1 The clustering results of the data field (category code + data field content) are as follows: 001;oxff,oxfe,oxfd,oxfd; 002;oxff,oxff,oxfd,oxfd; 003;oxff,oxfe,oxfe,oxfd.

[0068] 1.2 The timestamp processing result (category code + timestamp difference value) is as follows: 001,0;001,10;002,10;002,10;002,11;003,10;003,10;001,9;001,10;001,10;001,10;001,10。

[0069] The second step is to process the timestamp processing results from section 1.2: 2.1, Split the results from 1.2, placing the category code and timestamp difference values ​​into two separate arrays: Category code array: 001,001,002,002,002,003,003,001,001,001,001; Timestamp lookup array: 0,10,10,10,11,10,10,9,10,10,10.

[0070] 2.2, Perform a difference operation on the class code array from 2.1 to obtain the class code difference array: Category code difference array: 0,0,1,0,0,1,0,-2,0,0,0.

[0071] 2.3. Divide the timestamp difference array from 2.1 to obtain the timestamp difference array: Timestamp difference array: 0,10,10,10,11,10,10,9,10,10,10.

[0072] It is important to emphasize that the category code difference array and the timestamp difference array must correspond one-to-one; otherwise, subsequent decoding and restoration work cannot be performed.

[0073] 2.4 Store the category code difference array and the timestamp difference array in the following manner: For each difference value, count the number of times that difference value appears; The array corresponding to the category code difference array becomes: 0→2,1→1,0→2,1→1,0→1,-2→1,0→3; The timestamp difference array becomes: 0→1, 10→3, 11→1, 10→2, 9→1, 10→3.

[0074] The technical advantages of the above operation are: First, ensure the timing of all data, meaning that the order of all data can be restored, thus providing technical support for subsequent frequency reduction operations.

[0075] Secondly, in real-world IoT data, the periodicity and immutability of the data are very strong.

[0076] For example, if the category code (i.e. the content of the data field) remains unchanged, its lookup result will always be 0. If there are multiple repeated 0s, then you only need to store the number of 0s and consecutive 0s. 0→10 can represent 10 0s. If there is an original category code, then you need to store 10 repeated category codes. The storage volume will be greatly reduced by this method.

[0077] For example, if the class code changes periodically, i.e., the class code increases from 1, 2, 3, 4, ..., the difference result will always be 1. For multiple repeated 1s, you only need to store the 1 and the number of consecutive 1s, instead of storing the search value, which can greatly reduce the storage volume.

[0078] Secondly, see Figure 2 As shown in the figure, this application provides a time-series data processing apparatus, which includes: The data acquisition module is used to acquire multiple time-series data from the same data source in chronological order. The data comparison module is used to configure the corresponding category code based on the data field content of each time series data, obtain the timestamp deviation between each data and its previous time series data, and integrate them to generate the timestamp processing result corresponding to each time series data. The difference array generation module is used to obtain the corresponding category code difference array and timestamp difference array based on the timestamp processing results of each of the time series data. The difference array sorting module is used to separately count the different difference values ​​and corresponding consecutive occurrences in the category code difference array and the timestamp difference array, obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and integrate them to obtain the difference array information; The data processing module is used to integrate the difference array information, the timestamp information of the first time series data, and the data field content corresponding to each category code to obtain and store the processed time series data information; wherein, The same data field content corresponds to the same category code.

[0079] In this embodiment, the shortcomings of the prior art are overcome. The corresponding category code is configured according to the data field content of the time series data to classify the time series data. Then, the time series order is recorded by the timestamp deviation. By compressing and storing a large amount of time series data through the above key information, the time order and uniformity of data storage can be guaranteed. It supports fast down-sampling of data and ensures very high time series accuracy. The down-sampling data is smaller in size and saves more space.

[0080] Furthermore, the data comparison module is also used to obtain the corresponding category code for the data field content of each of the time series data; The data comparison module is also used to compare the timestamps of each of the time series data with the timestamps of the previous time series data to obtain the timestamp deviation between each of the time series data and the timestamps of the previous time series data. The data comparison module is also used to integrate the category code and timestamp deviation of each time series data into the corresponding timestamp processing result.

[0081] Furthermore, the difference array generation module is also used to extract the category code from the timestamp processing result of each of the time series data, compare it with the category code of the previous time series data, obtain the category code difference value corresponding to each of the time series data, and integrate them to obtain the category code difference array; The difference array generation module is also used to extract the timestamp deviation from the timestamp processing results of each of the time series data, and integrate them to obtain the timestamp difference array.

[0082] Furthermore, the difference array sorting module is also used to count the different difference values ​​and their corresponding consecutive occurrences in the category code difference array, associate each difference value with its corresponding consecutive occurrences to generate corresponding category code difference value-occurrence association data, and integrate each category code difference value-occurrence association data to obtain the category code difference merged encoding data. The difference array sorting module is also used to count the different difference values ​​and their corresponding consecutive occurrences in the timestamp difference array, associate each difference value with its corresponding consecutive occurrences to generate corresponding timestamp difference value-count association data, and integrate each timestamp difference value-count association data to obtain the timestamp difference merged encoding data. The differential array sorting module is also used to integrate differential array information based on the class code differential merged encoding data and the timestamp differential merged encoding data.

[0083] Furthermore, the device also includes: The data decoding module is used to extract the differential array information, the timestamp information of the first time series data, and the data field content corresponding to each category code from the information after processing the time series data, and decode to obtain multiple time series data from the same data source.

[0084] Furthermore, the device also includes: The data extraction module is used to obtain the corresponding category code differential merged encoding data and timestamp differential merged encoding data based on the differential array information in the time series data processed information, and decode to generate the corresponding category code differential data group and timestamp differential data group. The down-frequency processing module is used to sample the category code differential data group and the timestamp differential data group according to a preset down-frequency ratio to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. The data group processing module is used to locate position m in the first data in the category code sampling differential data group and the timestamp sampling differential data group, and add the difference value of each unsampled data before the m-th data in the category code sampling differential data group and the timestamp sampling differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; The data group processing module is further configured to, for each data item from the second to the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group, locate the (i-1)-th data item and the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group at their corresponding positions m and n in the category code differential data group and the timestamp differential data group, and add the difference value of the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group to the difference value of each unsampled data item between the m-th and n-th data items in the category code differential data group and the timestamp differential data group; The data group processing module is also used to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group to obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. The down-frequency acquisition module is used to decode and obtain multiple corresponding down-frequency sampled time series data based on the updated category code sampled differential data group and the updated timestamp sampled differential data group, combined with each category code and the corresponding data field content in the time series data processed information; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

[0085] In summary, the timing data processing apparatus provided in this application embodiment has the same technical principle as the timing data processing method provided in the first aspect in terms of technical problems, technical solutions and technical effects, so it will not be described again here.

[0086] Thirdly, this application provides an electronic device, the electronic device comprising: One or more processors; A storage device for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement the timing data processing method as mentioned in the first aspect.

[0087] Fourthly, this application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the timing data processing method mentioned in the first aspect.

[0088] In the description of this application, it should be noted that the terms "upper," "lower," etc., indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the system or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Unless otherwise expressly specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication between two components. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0089] It should be noted that in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0090] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A time-series data processing method, characterized in that, The method includes the following steps: Retrieve multiple time-series data from the same data source in chronological order; Based on the data field content of each time series data, configure their respective category codes, obtain the timestamp deviation between each and its previous time series data, and integrate to generate the timestamp processing result corresponding to each time series data; Based on the timestamp processing results of each of the time series data, the corresponding category code difference array and timestamp difference array are obtained; The different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array are counted respectively to obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and then integrated to obtain the difference array information; The differential array information, the timestamp information of the first time-series data, and the data field content corresponding to each category code are integrated to obtain and store the processed time-series data information; wherein... The same data field content corresponds to the same category code.

2. The time-series data processing method as described in claim 1, characterized in that, Based on the data field content of each time series data, configure the corresponding category code for each, and obtain the timestamp deviation between each and its previous time series data. Integrate and generate the timestamp processing result corresponding to each time series data, including the following steps: For each of the aforementioned time-series data, obtain the corresponding category code; Each of the time series data is compared with the time series data preceding it by timestamp to obtain the timestamp deviation between each of the time series data and the time series data preceding it; The category code and timestamp deviation of each of the aforementioned time series data are integrated into the corresponding timestamp processing result.

3. The time-series data processing method as described in claim 1, characterized in that, Based on the timestamp processing results of each of the aforementioned time-series data, the corresponding category code difference array and timestamp difference array are obtained, including the following steps: Extract the category code from the timestamp processing result of each time series data, compare it with the category code of the previous time series data, obtain the category code difference value corresponding to each time series data, and integrate them to obtain the category code difference array; Extract the timestamp deviation from the timestamp processing results of each of the time series data, and integrate them to obtain the timestamp difference array.

4. The time-series data processing method as described in claim 1, characterized in that, The different difference values ​​and their corresponding consecutive occurrences in the category code difference array and the timestamp difference array are counted separately to obtain the category code difference merged encoding data and the timestamp difference merged encoding data. These are then integrated to obtain the difference array information, including the following steps: The different difference values ​​and their corresponding consecutive occurrences in the category code difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding category code difference value-occurrence association data. The category code difference value-occurrence association data are then integrated to obtain the category code difference merged encoding data. The different difference values ​​and their corresponding consecutive occurrences in the timestamp difference array are statistically analyzed. Each difference value is associated with its corresponding consecutive occurrence, generating corresponding timestamp difference value-count association data. The timestamp difference value-count association data are then integrated to obtain the timestamp difference merged encoding data. Differential array information is obtained by integrating differentially merged encoding data based on category code and differentially merged encoding data based on timestamp.

5. The time-series data processing method as described in claim 1, characterized in that, The method further includes the following steps: Extract the differential array information, the timestamp information of the first time series data, and the data field content corresponding to each category code from the processed information of the time series data, and decode to obtain multiple time series data from the same data source.

6. The time-series data processing method as described in claim 1, characterized in that, The method further includes a down-frequency sampling process, which includes the following steps: Based on the differential array information in the processed time-series data, the corresponding category code differential merged encoding data and timestamp differential merged encoding data are obtained, and the corresponding category code differential data group and timestamp differential data group are decoded to generate the corresponding category code differential data group. According to the preset down-rate, the category code differential data group and the timestamp differential data group are sampled to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. For the first data in the category code sampling differential data group and the timestamp sampling differential data group, locate position m in the category code differential data group and the timestamp differential data group, and add the difference value of each unsampled data before the m-th data in the category code differential data group and the timestamp differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; Each data point from the second point onwards in the category code sampling differential data group and the timestamp sampling differential data group is referred to as the i-th data point. The positions m and n of the (i-1)-th and i-th data points in the category code sampling differential data group and the timestamp sampling differential data group are located in the category code differential data group and the timestamp differential data group, respectively. The difference value of the i-th data point in the category code sampling differential data group and the timestamp sampling differential data group is added to the difference value of each unsampled data point between the m-th and n-th data points in the category code differential data group and the timestamp differential data group. Repeat the above process to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group, and obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. Based on the updated category code sampling differential data group and the updated timestamp sampling differential data group, combined with each category code and the corresponding data field content in the time-series data processing information, multiple corresponding down-frequency sampling time-series data are decoded to obtain; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

7. A time-series data processing apparatus, characterized in that, The device includes: The data acquisition module is used to acquire multiple time-series data from the same data source in chronological order. The data comparison module is used to configure the corresponding category code based on the data field content of each time series data, obtain the timestamp deviation between each data and its previous time series data, and integrate them to generate the timestamp processing result corresponding to each time series data. The difference array generation module is used to obtain the corresponding category code difference array and timestamp difference array based on the timestamp processing results of each of the time series data. The difference array sorting module is used to separately count the different difference values ​​and corresponding consecutive occurrences in the category code difference array and the timestamp difference array, obtain the category code difference merged encoding data and the timestamp difference merged encoding data, and integrate them to obtain the difference array information; The data processing module is used to integrate the difference array information, the timestamp information of the first time series data, and the data field content corresponding to each category code to obtain and store the processed time series data information; wherein, The same data field content corresponds to the same category code.

8. The time-series data processing apparatus as described in claim 7, characterized in that, The device further includes: The data extraction module is used to obtain the corresponding category code differential merged encoding data and timestamp differential merged encoding data based on the differential array information in the time series data processed information, and decode to generate the corresponding category code differential data group and timestamp differential data group. The down-frequency processing module is used to sample the category code differential data group and the timestamp differential data group according to a preset down-frequency ratio to obtain the corresponding category code sampling differential data group and timestamp sampling differential data group. The data group processing module is used to locate position m in the first data in the category code sampling differential data group and the timestamp sampling differential data group, and add the difference value of each unsampled data before the m-th data in the category code sampling differential data group and the timestamp sampling differential data group to the first data in each of the category code sampling differential data group and the timestamp sampling differential data group; The data group processing module is further configured to, for each data item from the second to the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group, locate the (i-1)-th data item and the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group at their corresponding positions m and n in the category code differential data group and the timestamp differential data group, and add the difference value of the i-th data item in the category code sampling differential data group and the timestamp sampling differential data group to the difference value of each unsampled data item between the m-th and n-th data items in the category code differential data group and the timestamp differential data group; The data group processing module is also used to update the difference value of each data in the category code sampling difference data group and the timestamp sampling difference data group to obtain the updated category code sampling difference data group and the updated timestamp sampling difference data group. The down-frequency acquisition module is used to decode and obtain multiple corresponding down-frequency sampled time series data based on the updated category code sampled differential data group and the updated timestamp sampled differential data group, combined with each category code and the corresponding data field content in the time series data processed information; wherein... The ratio of the number of down-sampled timing data to the timing data corresponds to the down-sampling factor.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the timing data processing method as described in any one of claims 1 to 6.

10. A storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the timing data processing method as described in any one of claims 1 to 6.

Images