Timing data processing method and electronic device

Abstract

The application belongs to the technical field of computers, and particularly relates to a time series data processing method and an electronic device. The method comprises the following steps: acquiring a plurality of data sets with the same time series attribute, wherein the data set comprises a plurality of time series data with the time series attribute; sorting the time series data in the data set according to time series nodes to obtain a data sequence; determining a sequence length for accommodating the time series nodes according to the minimum time series node and the maximum time series node in the data sequence, and determining the number of binary bits for accommodating attribute values according to the maximum attribute value in the data sequence; initializing a sub-bitmap corresponding to the data set according to the minimum time series node, the sequence length and the number of binary bits, and writing the data sequence into the sub-bitmap; the sub-bitmap is an array for storing the data set in binary format; and merging the sub-bitmaps of the plurality of data sets to obtain a main bitmap corresponding to the time series attribute. The application can improve the storage space utilization rate of the time series data.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a time-series data processing method, a time-series data processing device, a computer-readable medium, an electronic device, and a computer program product. Background Technology

[0002] In data warehouse projects, data is typically extracted and calculated based on dates. This data changes continuously over time, and data processing often involves calculating daily characteristics or reviewing historical data for specific users. Date data is usually stored in structured data warehouses, requiring significant storage space. Summary of the Invention

[0003] This application provides a time-series data processing method, a time-series data processing apparatus, a computer-readable medium, an electronic device, and a computer program product, with the aim of improving the storage space utilization of time-series data.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of the embodiments of this application, a time-series data processing method is provided, the method comprising:

[0006] Obtain multiple data sets with the same time series attribute, the data set including multiple time series data with the time series attribute, the time series data including time series nodes and attribute values ​​associated with the time series nodes;

[0007] The time-series data in the dataset are sorted according to the time-series nodes to obtain a data sequence;

[0008] The sequence length to accommodate the time nodes is determined based on the minimum and maximum time nodes in the data sequence, and the number of binary bits to accommodate the attribute value is determined based on the maximum attribute value in the data sequence.

[0009] A sub-bitmap corresponding to the data set is initialized based on the minimum time sequence node, the sequence length, and the number of binary bits, and the data sequence is written into the sub-bitmap; the sub-bitmap is an array that stores the data set in binary format;

[0010] The sub-bitmaps of multiple data sets are merged to obtain the main bitmap corresponding to the time series attribute.

[0011] According to one aspect of the embodiments of this application, a time-series data processing apparatus is provided, the apparatus comprising:

[0012] The acquisition module is configured to acquire multiple data sets with the same time series attribute, the data set including multiple time series data with the time series attribute, the time series data including time series nodes and attribute values ​​associated with the time series nodes;

[0013] The sorting module is configured to sort the time-series data in the dataset according to the time-series nodes to obtain a data sequence.

[0014] The determining module is configured to determine the sequence length to accommodate the time nodes based on the minimum and maximum time nodes in the data sequence, and to determine the number of binary bits to accommodate the attribute value based on the maximum attribute value in the data sequence.

[0015] An initialization module is configured to initialize a sub-bitmap corresponding to the data set based on the minimum time sequence node, the sequence length, and the number of binary bits, and to write the data sequence into the sub-bitmap; the sub-bitmap is an array that stores the data set in binary format;

[0016] The merging module is configured to merge the sub-bitmaps of multiple data sets to obtain a master bitmap corresponding to the time-series attribute.

[0017] According to one aspect of the embodiments of this application, a computer-readable medium is provided, on which a computer program is stored, which, when executed by a processor, implements the timing data processing method as described in the above technical solutions.

[0018] According to one aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the executable instructions to implement the timing data processing method as described in the above technical solution.

[0019] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the timing data processing method as described in the above technical solutions.

[0020] In the technical solution provided in this application embodiment, by acquiring multiple data sets with the same temporal attributes, the temporal data in the data sets can be sorted according to temporal nodes to obtain a data sequence. Then, a sub-bitmap storing the data set in binary format is established for each data set based on the data sequence. Finally, the sub-bitmaps of multiple data sets are merged to obtain a main bitmap corresponding to the temporal attributes. This application embodiment can adaptively adjust the overall length of the bitmap according to the temporal data distribution of each data set, thus saving storage space for temporal data and improving the convenience of data expansion.

[0021] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0022] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0023] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0024] Figure 2 A flowchart of a timing data processing method according to one embodiment of this application is shown.

[0025] Figure 3 This paper illustrates a bitmap data structure provided in an application scenario according to an embodiment of this application.

[0026] Figure 4 The flowchart illustrates the sub-bitmap construction stage in an application scenario according to an embodiment of this application.

[0027] Figure 5 The flowchart illustrating an embodiment of this application for querying time-series data in an application scenario is shown.

[0028] Figure 6 The flowchart illustrating the addition of timing data in an application scenario according to an embodiment of this application is shown.

[0029] Figure 7 A schematic block diagram of the timing data processing apparatus provided in an embodiment of this application is shown.

[0030] Figure 8 A schematic diagram of a computer system architecture suitable for implementing the embodiments of this application is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0032] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0033] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] Figure 1 An exemplary system architecture block diagram illustrating the application of the technical solution of this application is shown schematically.

[0037] like Figure 1As shown, the system architecture 100 may include terminal devices 110, a network 120, and a server 130. Terminal devices 110 may include various electronic devices such as smartphones, tablets, laptops, desktop computer smart speakers, smart wearable devices, smart in-vehicle devices, and smart payment terminals. Server 130 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal devices 110 and server 130, such as a wired communication link or a wireless communication link.

[0038] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided in this application embodiment can be applied to terminal device 110, or to server 130, or can be implemented jointly by terminal device 110 and server 130. This application does not impose any special limitations on this.

[0039] For example, by installing applications on terminal device 110, various types of application services can be provided to users, such as instant messaging, VoIP, remote conferencing, online shopping, electronic payment, and so on. During the use of the corresponding application services, users will generate a large amount of time-series data, which can be stored locally on terminal device 110 or stored on server 130 via network 120.

[0040] Time-series data can be stored in a data warehouse, a strategic collection of all types of data supporting decision-making processes at all levels of an enterprise. It is a single data store created for analytical reporting and decision support purposes. For enterprises requiring business intelligence, it provides guidance for business process improvement, monitoring of time, cost, quality, and control.

[0041] In data warehouse projects, data is typically extracted and calculated based on dates. Data changes continuously over time. In data processing, there are often requirements to calculate daily characteristics or trace historical data for specific users. For example, for a particular user, the number of days or specific dates of certain events (such as purchases, being blacklisted, etc.) that occurred recently. Data tracing is also necessary; for instance, a model might output a score, and the user often needs to trace back the data, such as the score for a specific day within the last year. Blacklist systems need to know whether a user was on a blacklist on a specific date within the last year.

[0042] In the related technologies of this application, time-series data is generally stored in a data warehouse as structured data. As shown in Table 1 below, time-series data is stored using structured data storage and can be calculated using SQL.

[0043] Table 1

[0044] User ID date Corresponding value User 1 20210901 10 User 1 20210902 4 …… …… …… User 1 20220903 12 User 2 20210901 9 …… …… ……

[0045] Alternatively, data can be stored by concatenating dates horizontally, as shown in Table 2 below.

[0046] Table 2

[0047] User ID Date and corresponding value User 1 20210901:10,20210902:4,…… User 2 20210901:9 …… ……

[0048] Saving data in text format results in high storage costs due to the large number of bytes required for each date. Furthermore, calculations require parsing dates before statistical calculations, necessitating significant computing power and involving numerous repetitive calculations, leading to high computational costs.

[0049] To address the problems existing in the aforementioned related technologies, this application provides a scheme for generating time-series data storage structures, which can store the aforementioned date data using extremely low storage space. This scheme does not require a fixed start date and can determine the bitmap size corresponding to each date based on the numerical value, adaptively encoding and achieving automatic expansion, thus improving the flexibility of time-series data storage.

[0050] The following detailed description, in conjunction with specific embodiments, provides technical solutions such as time-series data processing methods, time-series data processing devices, computer-readable media, electronic devices, and computer program products provided in this application.

[0051] Figure 2 A flowchart of a time-series data processing method according to one embodiment of this application is shown. This time-series data processing method can be... Figure 1 The terminal device or server shown is used for execution. This application embodiment uses a timing data processing method executed by a terminal device as an example for illustration. For example... Figure 2As shown, the time-series data processing method may include the following steps S210 to S250.

[0052] S210: Obtain multiple data sets with the same time series attributes. The data sets include multiple time series data with time series attributes. The time series data includes time series nodes and attribute values ​​associated with the time series nodes.

[0053] S220: Sort the time-series data in the dataset according to the time-series nodes to obtain the data sequence.

[0054] S230: Determine the sequence length to accommodate the time nodes based on the minimum and maximum time nodes in the data sequence, and determine the number of binary bits to accommodate the attribute value based on the maximum attribute value in the data sequence.

[0055] S240: Initialize the sub-bitmap corresponding to the data set based on the minimum time sequence node, sequence length, and number of binary bits, and write the data sequence into the sub-bitmap; the sub-bitmap is an array that stores the data set in binary format.

[0056] S250: Merge the sub-bitmaps of multiple data sets to obtain the main bitmap corresponding to the time series attributes.

[0057] In the technical solution provided in this application embodiment, by acquiring multiple data sets with the same temporal attributes, the temporal data in the data sets can be sorted according to temporal nodes to obtain a data sequence. Then, a sub-bitmap storing the data set in binary format is established for each data set based on the data sequence. Finally, the sub-bitmaps of multiple data sets are merged to obtain a main bitmap corresponding to the temporal attributes. This application embodiment can adaptively adjust the overall length of the bitmap according to the temporal data distribution of each data set, thus saving storage space for temporal data and improving the convenience of data expansion.

[0058] The following provides a detailed description of the specific implementation methods of each step in this time-series data processing method based on multiple embodiments.

[0059] In step S210, multiple data sets with the same time series attributes are obtained. The data sets include multiple time series data with time series attributes. The time series data includes time series nodes and attribute values ​​associated with the time series nodes.

[0060] Time series data refers to data that is correlated with time series. Time series attributes represent the physical properties of the data in the actual application scenario, and time series nodes represent the nodes corresponding to the values ​​of each attribute in the time series.

[0061] For example, based on log data generated by a large number of users using a certain application, it is possible to obtain behavioral data of each user at different time granularities.

[0062] Taking an online trading platform as an example, the time-series attribute can be a user's platform login behavior, and the corresponding attribute value can be the number of times a user logs into the online trading platform each day; or, for example, the time-series attribute can be a user's online trading behavior, and the corresponding attribute value can be the number of times a user trades on the online trading platform each day.

[0063] In one embodiment of this application, before obtaining multiple data sets with the same time series attributes, a file containing time series data can be split into multiple data blocks; the time series data in the data blocks can be classified according to the time series attributes to obtain one or more data sets with time series attributes.

[0064] In this embodiment of the application, the time-series data in the file usually contains a variety of different time-series attributes. In order to improve the efficiency of time-series data processing, this embodiment of the application can first split the file to obtain multiple data blocks, and then classify the time-series data in each data block according to the time-series attributes to obtain multiple data sets with the same time-series attributes.

[0065] For example, the splitting of data blocks can be viewed as a random process, such as splitting according to file size, with each data block not exceeding 100MB. For each data block, the date data to be stored is extracted and categorized according to a specified classification dimension (e.g., by user ID), resulting in a data set corresponding to each category.

[0066] In step S220, the time-series data in the dataset are sorted according to the time-series nodes to obtain a data sequence.

[0067] Taking splitting a file into three data blocks as an example, we can obtain three data sets with the same temporal attributes. Sorting the three data sets according to their temporal nodes yields three corresponding data sequences, as shown in Tables 3, 4, and 5 below.

[0068] Table 3

[0069] date numerical values 2021-09-01 10 2021-09-02 4 2021-09-20 12 2021-09-21 13 2021-09-22 14 2021-09-23 15 2021-09-24 16 2021-09-25 17

[0070] Table 4

[0071] date numerical values 2021-09-04 10 2021-09-06 4 2021-09-07 12 2021-09-08 13 2021-09-09 14

[0072] Table 5

[0073]

[0074]

[0075] In step S230, the sequence length to accommodate the time nodes is determined based on the minimum and maximum time nodes in the data sequence, and the number of binary bits to accommodate the attribute value is determined based on the maximum attribute value in the data sequence.

[0076] Taking Table 3 as an example, the smallest time-series node in this data sequence is 2021-09-01, and the largest time-series node is 2021-09-25. Therefore, the sequence length used to accommodate these time-series nodes is 25. The largest attribute value in this data sequence is 17. Therefore, the number of binary bits used to accommodate these attribute values ​​is 5, meaning that the maximum value range that can be accommodated using 5 binary bits is 0-31.

[0077] In one embodiment of this application, before determining the sequence length to accommodate the time-series nodes based on the minimum and maximum time-series nodes in the data sequence, a continuity detection can be performed on the time-series nodes in the data sequence to obtain missing nodes; the missing nodes and specified attribute values ​​associated with the missing nodes are then filled into the data sequence.

[0078] In one embodiment of this application, a method for detecting missing nodes by performing continuity detection on time-series nodes in a data sequence may include: sequentially obtaining the time difference between two adjacent time-series nodes in the data sequence; when the time difference is greater than one, determining the missing node between the two adjacent time-series nodes based on the time difference.

[0079] For example, in Table 2, the second time series node is 2021-09-02 and the third time series node is 2021-09-20. The time series difference between the two is greater than one, which indicates that there is a missing node between them, that is, the data from September 3 to September 19 is missing.

[0080] This application embodiment can fill in the missing nodes in the data sequence, and can fill in the missing nodes with specified attribute values ​​associated with the missing nodes. For example, the specified attribute value can be 0, -1, or any other value that does not conflict with the attribute values ​​in the data set.

[0081] This application embodiment fills missing nodes into the data sequence, which can reserve storage space for missing nodes in the sub-bitmap, making it convenient to directly update the values ​​corresponding to the missing nodes during the subsequent merging of sub-bitmaps, without having to update all attribute values.

[0082] In step S240, a sub-bitmap corresponding to the data set is initialized based on the minimum time node, sequence length, and number of binary bits, and the data sequence is written into the sub-bitmap; the sub-bitmap is an array that stores the data set in binary format.

[0083] In one embodiment of this application, the sub-bitmap includes a plurality of continuously distributed description fields and a data segment. The description fields include a node start field, a node number field, and a node length field.

[0084] The data segment is used to store the attribute values ​​associated with each time-series node in binary format. The node start field is used to indicate the time-series node associated with the first attribute value stored in the data segment. The node number field is used to indicate the total number of time-series nodes associated with all attribute values ​​stored in the data segment. The node length field is used to indicate the number of bits of binary value occupied by each attribute value stored in the data segment.

[0085] Figure 3 This illustration shows a bitmap data structure provided by an embodiment of this application in an application scenario. For example... Figure 3 As shown, the bitmap data structure includes the following multiple data segments.

[0086] The node start field, also known as the start date (S), refers to the number of days between a fixed date D and the start date. This field is Bs bits long, so the start date can represent a time range of [D, D+2^Bs). For example, if D is 2010-01-01 and Bs = 15, then 2^15 = 32768 days, and the start date (S) can represent a time range of [2010-01-01, 2099 / 09 / 19].

[0087] The number of nodes field, or days (M), indicates how long the data needs to be stored starting from the start date S. It's stored as a single-digit number (Bm), meaning the maximum number of days to store is 2^Bm. For example, if Bm is set to 10 digits, the maximum number of days is 1024.

[0088] The node length field, or digits occupied per day (T), represents the number of digits used to store the value each day. Since the number of days is the same, this value depends on the largest value among all days. Here, we use Bt digits to store the value, so the range of digits occupied is [1-2^Bt], and the range of the represented value is [0-2^(2^Bt)]. For example, if Bt is 4 digits, then the range of digits occupied is [1-16]. When the maximum digit occupied is 16, it can represent a value range of [0, 65536) (i.e., 2^(2^4) = 2^16 = 65536).

[0089] Reserved field: that is, reserved number of bits (R), for future expansion.

[0090] Data segment: This is the list of bits corresponding to each day, with a length of M*T. Bits 1 to M store the value corresponding to the start date S; bits M+1 to 2M store the value corresponding to the start date S+1 day, and so on.

[0091] In one embodiment of this application, initializing a sub-bitmap corresponding to a data set based on the minimum time sequence node, sequence length, and number of binary bits includes: obtaining the time difference between the minimum time sequence node and a preset fixed time sequence node, and assigning a value to the node start field based on the time difference; assigning a value to the node number field based on the sequence length; and assigning a value to the node length field based on the number of binary bits.

[0092] In one embodiment of this application, writing a data sequence into a sub-bitmap includes: converting each attribute value in the data sequence into a binary value with a length of binary bits; and writing the binary values ​​into the data segment of the sub-bitmap in the order of the timing nodes in the data sequence.

[0093] In step S250, the sub-bitmaps of multiple data sets are merged to obtain the main bitmap corresponding to the time series attributes.

[0094] In one embodiment of this application, merging sub-bitmaps of multiple data sets to obtain a master bitmap corresponding to a time-series attribute includes: determining the time-series interval of the sub-bitmaps of the data sets based on the node start field and the node count field; if the time-series interval of the first sub-bitmap includes the time-series interval of the second sub-bitmap, then filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap; if the time-series intervals of the first sub-bitmap and the second sub-bitmap do not have an inclusion relationship, then updating the node start field and the node count field of the first sub-bitmap based on the node start field and the node count field of the second sub-bitmap, and filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap; when the sub-bitmaps of multiple data sets are all merged into the first sub-bitmap, the first sub-bitmap is used as the master bitmap corresponding to the time-series attribute.

[0095] In this embodiment, sub-bitmaps are merged pairwise to obtain a master bitmap. When the time intervals of two sub-bitmaps have an inclusion relationship, the sub-bitmap with the smaller time interval can be merged into the sub-bitmap with the larger time interval, thereby reducing the amount of data computation and improving data processing efficiency.

[0096] In one embodiment of this application, before filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap, if the node length field of the first sub-bitmap is less than the node length field of the second sub-bitmap, then the node length field and data segment of the first sub-bitmap are updated according to the node length field of the second sub-bitmap.

[0097] When it is necessary to merge a sub-bitmap with a larger node length field into a sub-bitmap with a smaller node length field, the node length field and the value of each attribute stored in the data segment can be updated adaptively to achieve automatic bitmap expansion.

[0098] The implementation of the time-series data processing method provided in the embodiments of this application will be described below in conjunction with specific application scenarios.

[0099] In specific application scenarios, Hive's UDAF (User Defined Aggregate Function) approach is used, constructed within a distributed environment. Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It's a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop. Hive data warehouse tools can map structured data files to database tables and provide SQL query functionality, transforming SQL statements into distributed tasks for execution. UDAF refers to user-defined aggregate functions.

[0100] In Hive, SQL execution is ultimately translated into MapReduce for execution. The corresponding UDAF includes the sub-bitmap construction phase and the sub-bitmap merging phase, which are briefly explained below.

[0101] During the sub-bitmap construction phase, the Iterate method processes the column parameters passed in by the UDAF one by one, which in this case refers to (date + value). At the end of the sub-bitmap construction phase, the terminatePartial method is executed to process the data from the sub-bitmap construction phase and finally output it.

[0102] During the sub-bitmap merging phase, the merge method is used (to globally merge the locally summarized data) to merge the output results of all sub-bitmap construction phases.

[0103] Since data file splitting during data storage is independent of user ID, data for the same user will be split into multiple data blocks. Therefore, at this stage, the data in each data block needs to be partially summarized first (since the data in the same data block may be too large, to avoid excessive memory consumption, this solution summarizes every 6 data blocks, and then summarizes again at the end of the sub-bitmap construction stage). Then, in the sub-bitmap merging stage, data with the same user ID will be summarized together, and then a global summary will be performed.

[0104] Figure 4 A flowchart illustrating the sub-bitmap construction stage in an application scenario according to an embodiment of this application is shown. For example... Figure 4 As shown, the process of constructing a sub-bitmap for each data set includes the following steps S401 to S411.

[0105] S401: For a single user, add the data <date, value> sequentially to the list.

[0106] Use a list to store the data. For example, for the data sequence shown in Table 3 above, each (date, value) can be added to the list in sequence.

[0107] S402: Determine if the list contains at least 6 items.

[0108] If the list contains fewer than 6 items, return to step S401 and continue adding data to the list. If the list contains 6 items, proceed to step S403.

[0109] S403: Determine whether the sub-bitmap has been initialized.

[0110] If the sub-bitmap has not yet been initialized, proceed to step S404. If the sub-bitmap has been initialized, proceed to step S408.

[0111] S404: Determine the number of digits Bt occupied per day based on the maximum value in the list.

[0112] Suppose that the data in the list at this time is [(2021-09-01,10),(2021-09-02,4),(2021-09-20,12),(2021-09-21,13),(2021-09-22,14),(2021-09-23,15)].

[0113] The maximum value is 15, from which we can calculate that the number of digits occupied per day, Bt, is 4, indicating that the maximum value range is 0-15.

[0114] S405: Determine the start date S and the number of days M based on the maximum and minimum dates in the list.

[0115] Based on the data in the list, the starting date S is (2021-09-01), the maximum date is (2021-09-23), and the number of days M is 23.

[0116] S406: Initialize the sub-bitmap data structure based on Bt, S, and M.

[0117] Based on the above data, the initialized sub-bitmap data structure can be represented as DayBit(2021-09-01,23,4).

[0118] S407: Add the values ​​from the list to the sub-map and clear the list.

[0119] After clearing the list, return to step S401 and continue adding data to the list.

[0120] For example, add the remaining date data to the list, and the data in the list will be [(2021-09-24,16),(2021-09-25,17)].

[0121] S408: Determine the number of bits Bt occupied per day based on the maximum value in the list and the original DayBit.

[0122] Comparing the data in the list with DayBit(2021-09-01,23,4), we determine that the maximum value is 17, and thus we can determine that the number of bits occupied per day, Bt, is 5, which means that the maximum value range is 0-31.

[0123] S409: Determine the start date S and the number of days M based on the maximum date, minimum date, and original DayBit ​​in the list.

[0124] Compare the data in the list with DayBit(2021-09-01,23,4) to determine the start date S as (2021-09-01) and the maximum number of days as (2021-09-25), then the number of days M is 25.

[0125] S410: Initialize a new sub-bitmap data structure based on the updated Bt, S, and M.

[0126] S411: Copy the original DayBit ​​data to the new DayBit ​​and discard the original DayBit.

[0127] Initialize a new DayBit ​​(2021-09-01,25,5) and copy the data from the old DayBit ​​into it. Then continue to execute step S407, adding the data [(2021-09-24,16),(2021-09-25,17)] from the list into the new DayBit.

[0128] based on Figure 4 The sub-bitmap construction method shown above processes the three data sequences shown in Tables 3, 4 and 5 with time-series data to obtain the corresponding three sub-bitmaps DayBit(2021-09-01,25,5), DayBit(2021-09-04,6,4) and Daybit(2021-08-29,40,8).

[0129] During the sub-bitmap merging phase, the first two sub-bitmaps, DayBit(2021-09-01,25,5) and DayBit(2021-09-04,6,4), can be merged. The date in the first sub-bitmap, DayBit(2021-09-01,25,5), includes the date in the second sub-bitmap, DayBit(2021-09-04,6,4), and the number of digits for each day's value in the first sub-bitmap is greater than that in the second sub-bitmap. Therefore, the date values ​​from the second sub-bitmap can be used to fill the first sub-bitmap.

[0130] Next, comparing the first sub-bitmap DayBit ​​(2021-09-01, 25, 5) and the third sub-bitmap Daybit (2021-08-29, 40, 8), it can be seen that the date in the third sub-bitmap includes the first sub-bitmap, and the number of digits for each day in the third sub-bitmap is greater than that in the first sub-bitmap. Therefore, the date values ​​from the first sub-bitmap can be filled into the third sub-bitmap, ultimately obtaining the main bitmap Daybit (2021-08-29, 40, 8). After obtaining the main bitmap, it can be further encoded and stored using Base64.

[0131] Figure 5 This document illustrates a flowchart of a time-series data query process in an application scenario, based on an embodiment of this application. For example... Figure 5 As shown, the process of querying time series data through the main bitmap includes the following steps S501 to S506.

[0132] S501: Read the Daybit data from the main bitmap.

[0133] S502: Parse fields Bs, Bm, and Bt respectively to obtain the start date, total number of days, and number of bits occupied T.

[0134] S503: Input the query date and calculate the difference in days between the query date and the start date.

[0135] S504: Determine whether the difference in the number of days is greater than or equal to 0 and less than the total number of days.

[0136] If the difference in the number of days is greater than or equal to 0 and less than the total number of days, then proceed to step S505. Otherwise, return an empty query result.

[0137] S505: Calculate the starting bit corresponding to the query date.

[0138] S506: Read T bits from the starting bit position, convert the result into a number and return the query result.

[0139] Figure 6 This document illustrates a flowchart of an embodiment of the present application for adding timing data in an application scenario. For example... Figure 6 As shown, the process of adding time series data to the main bitmap includes the following steps S601 to S612.

[0140] S601: Read the raw Daybit data.

[0141] S602: Parse fields Bs, Bm, and Bt respectively to obtain the start date, total number of days, and number of bits occupied, T.

[0142] S603: Calculate the number of days from the current date to the start date.

[0143] S604: Calculate the number of bits T' required for the current value based on the value of the day.

[0144] S605: Determine whether the number of days is less than 0 or greater than the total number of days.

[0145] If the number of days is less than 0 or greater than the total number of days, proceed to step S606. Otherwise, proceed to step S611.

[0146] S606: Determine whether the number of bits occupied T' is less than or equal to the number of bits occupied T.

[0147] If the number of bits occupied T' is less than or equal to the number of bits occupied T, then proceed to step S607. Otherwise, proceed to step S611.

[0148] S607: Recalculate Bs and Bm to generate a new Daybit.

[0149] S608: Uses bitmap batch copying to migrate old Daybit data to new Daybit.

[0150] Jump to step S611.

[0151] S609: Recalculate Bs, Bm, and Bt to generate a new Daybit.

[0152] S610: Performs data copying on a single date, from the start date to the end date of the old Daybit, migrating the old Daybit data to the new Daybit.

[0153] S611: Get the bit position corresponding to the current date.

[0154] S612: Write the current value to the bit position corresponding to the current date.

[0155] Based on the above application scenarios, it is clear that storing time-series data using a Daybit master bitmap can save storage space. Assuming there are M consecutive dates, and each date's value occupies at most T bits, the total length of the Daybit master bitmap in bits can be represented as Bs + Bm + Bt + Br + M*T.

[0156] Assuming the ID is 32 characters long (8 bits per character), the number of days is 180, the value ranges from 0 to 127 (8 bits, or 3 characters if stored as a string), and each date occupies 8 characters, then using the storage method shown in Table 1, the space used by a single user is: 180 (days) * (32 * 8 (ID space) + 8 * 8 (date space) + 3 * 8 (value space)) = 180 (days) * 344 (bits) = 61920 (bits).

[0157] If the horizontal splicing method shown in Table 2 is used for storage, then the space used is: 32*8 (ID occupied) + 180 (days) * (8*8 (date occupied) + 3*8 (value occupied)) = 256 + 180 (days) * 88 (bits) = 16096 (bits).

[0158] If the scheme provided in the embodiments of this application is used, then Bs=15, Bm=10, Bt=3, Br is a reserved field (assumed to be 3 here), M=180, T=7, then the space used is: 32*8+(15+10+3+3+180*7)=1547 (bits).

[0159] Therefore, compared with the storage method shown in Table 2, the storage space of this application is reduced by 97.5%. Compared with the storage method shown in Table 3, the storage space of this application is reduced by 90.38%.

[0160] Using different dates and number of days as examples, the space saved by this application compared to the storage methods of related technologies is shown in Table 6 below.

[0161] Table 6

[0162]

[0163]

[0164] Based on the above comparison, it can be seen that this application can improve the storage space compression rate of time series data, and the use of string processing method for time series data processing reduces a large amount of computation.

[0165] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0166] The following describes an apparatus embodiment of this application, which can be used to execute the timing data processing method in the above embodiments of this application. Figure 7 A schematic block diagram of a timing data processing apparatus provided in an embodiment of this application is shown. Figure 7 As shown, the time-series data processing apparatus 700 includes:

[0167] The acquisition module 710 is configured to acquire multiple data sets having the same time series attribute, the data set including multiple time series data having the time series attribute, the time series data including time series nodes and attribute values ​​associated with the time series nodes;

[0168] The sorting module 720 is configured to sort the time-series data in the dataset according to the time-series nodes to obtain a data sequence.

[0169] The determining module 730 is configured to determine the sequence length for accommodating the time nodes based on the minimum and maximum time nodes in the data sequence, and to determine the number of binary bits for accommodating the attribute value based on the maximum attribute value in the data sequence.

[0170] Initialization module 740 is configured to initialize a sub-bitmap corresponding to the data set based on the minimum timing node, the sequence length, and the number of binary bits, and write the data sequence into the sub-bitmap; the sub-bitmap is an array that stores the data set in binary format;

[0171] The merging module 750 is configured to merge the sub-bitmaps of multiple data sets to obtain a main bitmap corresponding to the time series attribute.

[0172] In some embodiments of this application, based on the above embodiments, the sub-bitmap includes a plurality of continuously distributed description fields and a data segment. The description fields include a node start field, a node quantity field, and a node length field. The data segment is used to store attribute values ​​associated with each of the time-series nodes in binary format. The node start field is used to indicate the time-series node associated with the first attribute value stored in the data segment. The node quantity field is used to indicate the total number of time-series nodes associated with all attribute values ​​stored in the data segment. The node length field is used to indicate the number of bits in the binary value occupied by each attribute value stored in the data segment.

[0173] In some embodiments of this application, based on the above embodiments, the initialization module 740 may further include:

[0174] The first assignment module is configured to obtain the time difference between the minimum timing node and the preset fixed timing node, and assign the node start field a value based on the time difference.

[0175] The second assignment module is configured to assign a value to the node count field based on the sequence length;

[0176] The third assignment module is configured to assign a value to the node length field based on the number of binary bits.

[0177] In some embodiments of this application, based on the above embodiments, the initialization module 740 further includes:

[0178] The conversion module is configured to convert each attribute value in the data sequence into a binary value with a length of the number of binary bits.

[0179] The writing module is configured to write the binary values ​​into the data segment of the sub-bitmap sequentially according to the arrangement order of each time node in the data sequence.

[0180] In some embodiments of this application, based on the above embodiments, the merging module 750 includes:

[0181] The interval determination module is configured to determine the temporal interval of the sub-bitmap of the data set based on the node start field and the node number field;

[0182] The first filling module is configured to fill the data segment of the first sub-bitmap into the data segment of the first sub-bitmap if the timing interval of the first sub-bitmap includes the timing interval of the second sub-bitmap.

[0183] The second filling module is configured to update the node start field and node number field of the first sub-bitmap according to the node start field and node number field of the second sub-bitmap if the time intervals of the first sub-bitmap and the second sub-bitmap do not have an inclusion relationship, and fill the data segment of the second sub-bitmap into the data segment of the first sub-bitmap.

[0184] The primary bitmap determination module is configured to use the first sub-bitmap as the primary bitmap corresponding to the time series attribute when multiple sub-bitmaps of the data sets are merged into the first sub-bitmap.

[0185] In some embodiments of this application, based on the above embodiments, the initialization module 740 further includes:

[0186] The field update module is configured to update the node length field and data segment of the first sub-bitmap according to the node length field of the second sub-bitmap if the node length field of the first sub-bitmap is less than the node length field of the second sub-bitmap.

[0187] In some embodiments of this application, based on the above embodiments, the sorting module 720 is further configured to: perform continuity detection on the time-series nodes in the data sequence to obtain missing nodes; and fill the data sequence with the missing nodes and specified attribute values ​​associated with the missing nodes.

[0188] In some embodiments of this application, based on the above embodiments, the sorting module 720 is further configured to: sequentially obtain the time difference between two adjacent time nodes in the data sequence; when the time difference is greater than one, determine the missing node between the two adjacent time nodes based on the time difference.

[0189] In some embodiments of this application, based on the above embodiments, the acquisition module is further configured to: split the file containing time-series data into multiple data blocks; classify the time-series data in the data blocks according to time-series attributes to obtain one or more data sets having the time-series attributes.

[0190] The specific details of the timing data processing apparatus provided in the various embodiments of this application have been described in detail in the corresponding method embodiments, and will not be repeated here.

[0191] Figure 8 A schematic block diagram of a computer system architecture for implementing an electronic device according to embodiments of the present application is shown.

[0192] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0193] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM). The random access memory 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output interface 805 (I / O interface) is also connected to the bus 804.

[0194] The following components are connected to the input / output interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a local area network card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 810 as needed so that computer programs read from it can be installed into the storage section 808 as needed.

[0195] Specifically, according to embodiments of this application, the processes described in the various method flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit 801, it performs various functions defined in the system of this application.

[0196] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0197] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0198] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0199] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.

[0200] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0201] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A time-series data processing method, characterized in that, include: Obtain multiple data sets with the same time series attribute, the data set including multiple time series data with the time series attribute, the time series data including time series nodes and attribute values ​​associated with the time series nodes; The time-series data in the dataset are sorted according to the time-series nodes to obtain a data sequence; The sequence length to accommodate the time nodes is determined based on the minimum and maximum time nodes in the data sequence, and the number of binary bits to accommodate the attribute value is determined based on the maximum attribute value in the data sequence. Initialize a sub-bitmap corresponding to the data set based on the minimum time sequence node, the sequence length, and the number of binary bits, and write the data sequence into the sub-bitmap; The sub-bitmap is an array that stores the data set in binary format; The sub-bitmaps of multiple data sets are merged to obtain the main bitmap corresponding to the time series attribute; The sub-bitmap includes multiple continuously distributed description fields and a data segment. The description fields include a node start field, a node count field, and a node length field. The data segment stores attribute values ​​associated with each of the time-series nodes in binary format. The node start field indicates the time-series node associated with the first attribute value stored in the data segment. The node count field indicates the total number of time-series nodes associated with all attribute values ​​stored in the data segment. The node length field indicates the number of bits in the binary value occupied by each attribute value stored in the data segment. The step of initializing the sub-bitmap corresponding to the data set based on the minimum time-series node, the sequence length, and the number of binary bits includes: obtaining the time difference between the minimum time-series node and a preset fixed time-series node, and assigning a value to the node start field based on the time difference; assigning a value to the node quantity field based on the sequence length; and assigning a value to the node length field based on the number of binary bits. The step of writing the data sequence into the sub-bitmap includes: converting each attribute value in the data sequence into a binary value with a length equal to the number of binary bits; and writing the binary values ​​into the data segment of the sub-bitmap in the order of the time sequence nodes in the data sequence. The step of merging multiple sub-bitmaps of the data sets to obtain a master bitmap corresponding to the time-series attribute includes: determining the time-series interval of the sub-bitmaps of the data sets based on the node start field and the node count field; if the time-series interval of the first sub-bitmap includes the time-series interval of the second sub-bitmap, then filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap; if the time-series intervals of the first sub-bitmap and the second sub-bitmap do not have an inclusion relationship, then updating the node start field and the node count field of the first sub-bitmap based on the node start field and the node count field of the second sub-bitmap, and filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap; when multiple sub-bitmaps of the data sets are all merged into the first sub-bitmap, the first sub-bitmap is used as the master bitmap corresponding to the time-series attribute.

2. The time-series data processing method according to claim 1, characterized in that, Before filling the data segment of the second sub-bitmap into the data segment of the first sub-bitmap, the method further includes: If the node length field of the first sub-bitmap is less than the node length field of the second sub-bitmap, then the node length field and data segment of the first sub-bitmap are updated according to the node length field of the second sub-bitmap.

3. The time-series data processing method according to claim 1 or 2, characterized in that, Before determining the sequence length to accommodate the time nodes based on the minimum and maximum time nodes in the data sequence, the method further includes: Missing nodes are obtained by performing continuity detection on the temporal nodes in the data sequence; Fill the data sequence with the missing nodes and the specified attribute values ​​associated with the missing nodes.

4. The time-series data processing method according to claim 3, characterized in that, To identify missing nodes by performing continuity detection on time-series nodes in the data sequence, the following steps are included: The timing difference between two adjacent time nodes in the data sequence is obtained sequentially; When the timing difference is greater than one, the missing node between the two adjacent timing nodes is determined based on the timing difference.

5. The time-series data processing method according to claim 1 or 2, characterized in that, Before acquiring multiple data sets with the same temporal attributes, the method further includes: Split the file containing time-series data into multiple data blocks; The time-series data in the data block are classified according to their time-series attributes to obtain one or more data sets with the time-series attributes.

6. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the timing data processing method as described in any one of claims 1 to 5.

Images