Data index construction method and device, equipment and storage medium
By determining the time granularity setting rules in the data index method, predicting the data attribute distribution and building an initial index, the problems of slowing data query speed and increasing movement overhead during data insertion are solved, and efficient data processing and system performance improvement are achieved.
Patent Information
- Application Number
- CN202411868747.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-03-28
AI Technical Summary
The existing data indexing methods slow down during data query, and the movement overhead increases when data is inserted, especially when facing massive data.
By determining the time granularity setting rules for the target time series data, predict the data attribute distribution within the current time granularity, build an initial index, and update the index after the time point exceeds the current time granularity to optimize the data query and insertion process.
It improves the speed of data query, reduces the movement overhead during data insertion, significantly improves data processing efficiency, and ensures the overall performance of the system.
Smart Images
Figure CN119719111A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data index technology, and in particular to a data index construction method, device, equipment and storage medium. Background Art
[0002] Existing indexing methods often need to traverse a large number of nodes or buckets during query operations. This linear or near-linear search process will cause a significant decrease in query speed when the amount of data is large. In addition, existing methods do not provide sufficient support for multi-dimensional queries and cannot effectively utilize the periodic characteristics of time series data for optimization. Moreover, since existing indexing methods lack the prediction and optimization of data attribute distribution when inserting data, operations are redundant during the insertion process, and the node structure needs to be adjusted frequently or the hash needs to be recalculated, resulting in higher mobile overhead. When faced with massive amounts of data, this mobile overhead not only affects the insertion efficiency, but also causes the overall performance of the system to decline.
[0003] In summary, how to solve the problem of slowing down data query by using data indexes and increasing movement overhead when inserting data is a technical problem that needs to be solved urgently. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a data index construction method, device, equipment and storage medium, which can solve the problem of slowing down the data query speed and increasing the movement overhead when inserting data using data index. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a data index construction method, comprising:
[0006] Determine the time granularity setting rules corresponding to the target time series data;
[0007] Setting the current time granularity corresponding to the current time point according to the time granularity setting rule, and determining the historical time granularity corresponding to the current time granularity;
[0008] Predicting a first target data attribute distribution of the target time series data within the current time granularity by using the historical data attributes of the historical time series data within the historical time granularity, and constructing a first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index;
[0009] After the current time point exceeds the time period corresponding to the current time granularity, a second target data attribute distribution of the target time series data is determined, and the second initial index of the target time series data is updated based on the second target data attribute distribution to obtain a target index of the target time series data.
[0010] Optionally, setting the current time granularity corresponding to the current time point according to the time granularity setting rule, and determining the historical time granularity corresponding to the current time granularity includes:
[0011] According to the time granularity setting rule, a first time granularity corresponding to the current time point and a second time granularity within the first time granularity are set, and the second time granularity is used as the current time granularity;
[0012] A third time granularity corresponding to the first time granularity and before the current time point is determined, and a fourth time granularity within the third time granularity and corresponding to the current time granularity is determined, so as to use the fourth time granularity as the historical time granularity.
[0013] Optionally, the predicting the first target data attribute distribution of the target time series data within the current time granularity by using the historical data attribute of the historical time series data within the historical time granularity includes:
[0014] Determine, according to a preset cumulative probability function, a first cumulative probability corresponding to the historical data attribute of the historical time series data within the historical time granularity;
[0015] Based on the first cumulative probability, learning index fitting is performed to obtain a historical position function corresponding to the historical time series data, and based on the historical position function, the first target data attribute distribution of the target time series data within the current time granularity is predicted; the historical position function is used to characterize the mapping relationship between the historical time series data and the historical index corresponding to the historical time series data.
[0016] Optionally, the predicting the first target data attribute distribution of the target time series data within the current time granularity based on the historical position function includes:
[0017] Increasing the slope of the historical position function to obtain a first target position function, and predicting the first target data attribute distribution of the target time series data within the current time granularity based on the first target position function;
[0018] Accordingly, constructing a first initial index of the target time series data based on the first target data attribute distribution includes:
[0019] The first initial index of the target time series data is constructed based on the first target data attribute distribution and a preset gap array.
[0020] Optionally, determining a second target data attribute distribution of the target time series data, and updating the second initial index of the target time series data based on the second target data attribute distribution to obtain a target index of the target time series data includes:
[0021] Determine, according to the preset cumulative probability function, a second cumulative probability corresponding to the data attribute value of the target time series data corresponding to the second initial index;
[0022] Performing learning index fitting based on the second cumulative probability to obtain a second target position function corresponding to the target time series data corresponding to the second initial index, and determining the corresponding second target data attribute distribution based on the second target position function;
[0023] The second initial index of the target time series data is updated based on the second target data attribute distribution to obtain a target index of the target time series data.
[0024] Optionally, the data index construction method further includes:
[0025] Determine the query time period and the query attribute value range of the preset query condition;
[0026] Determine the corresponding target time granularity according to the time period to be queried, and determine the time series data group corresponding to the target time granularity; wherein each of the time series data groups includes the target time series data within the corresponding target time granularity;
[0027] Each of the time series data groups is traversed based on the target index and the attribute value range to be queried to obtain target query data.
[0028] Optionally, after traversing each of the time series data groups based on the target index and the range of attribute values to be queried to obtain target query data, the method further includes:
[0029] The target query data corresponding to the minimum attribute value within the to-be-queried attribute value range in each of the time series data groups is stored in a preset priority queue;
[0030] The target query data is merged and sorted using the preset priority queue according to the attribute value of the target query data to obtain the sorted target query data.
[0031] In a second aspect, the present application provides a data index construction device, comprising:
[0032] A rule determination module is used to determine the time granularity setting rule corresponding to the target time series data;
[0033] A time granularity determination module, used to set the current time granularity corresponding to the current time point according to the time granularity setting rule, and determine the historical time granularity corresponding to the current time granularity;
[0034] An index building module, used to predict a first target data attribute distribution of the target time series data within the current time granularity by using the historical data attributes of the historical time series data within the historical time granularity, and to build a first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index;
[0035] An index updating module is used to determine a second target data attribute distribution of the target time series data after the current time point exceeds the time period corresponding to the current time granularity, and to update the second initial index of the target time series data based on the second target data attribute distribution to obtain a target index of the target time series data.
[0036] In a third aspect, the present application provides an electronic device, including:
[0037] Memory, used to store computer programs;
[0038] The processor is used to execute the computer program to implement the aforementioned data index construction method.
[0039] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned data index construction method is implemented.
[0040] In the present application, firstly, a time granularity setting rule corresponding to the target time series data is determined; then, the current time granularity corresponding to the current time point is set according to the time granularity setting rule, and the historical time granularity corresponding to the current time granularity is determined; then, the data attribute of the historical time series data within the historical time granularity is used to predict the first target data attribute of the target time series data within the current time granularity, and an initial index of the target time series data is constructed based on the first target data attribute; finally, after the current time point exceeds the time period corresponding to the current time granularity, the second target data attribute of the target time series data is determined, and the initial index of the target time series data is updated based on the second target data attribute to obtain the target index of the target time series data. As can be seen from the above, the present application sets a time granularity setting rule for time series data, and groups the target time series data by setting an appropriate time granularity. In this way, the efficiency of data processing can be improved; then, based on the data attributes of the historical target time series data, the initial index of the target time series data is constructed, and the index space is pre-allocated. In this way, the mobile overhead when the target time series data is inserted can be reduced, and the data processing efficiency can be significantly improved; then, the initial index of the target time series data is updated using the data attributes of the target time series data in a stable time period to obtain the target index. When querying data, the required data can be quickly and accurately found based on the time granularity, attributes and target index. In this way, the present application can solve the problem of slowing down the speed of data query and increasing the mobile overhead when inserting data by using data index through the above process. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0042] Figure 1 A flow chart of a data index construction method provided in this application;
[0043] Figure 2 A schematic diagram of data grouping based on time granularity provided by this application;
[0044] Figure 3 A schematic diagram of initial index construction provided for this application;
[0045] Figure 4 A schematic diagram of constructing a target index provided for this application;
[0046] Figure 5 A flow chart of a specific data index construction method provided in this application;
[0047] Figure 6 A data query schematic diagram provided for this application;
[0048] Figure 7 A specific data query flow chart provided for this application;
[0049] Figure 8 A schematic diagram of the structure of a data index construction device provided in this application;
[0050] Fig. 9 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION
[0051] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0052] Existing indexing methods often need to traverse a large number of nodes or buckets during query operations. This linear or near-linear search process will cause a significant decrease in query speed when the amount of data is large. In addition, existing methods do not provide sufficient support for multidimensional queries and cannot effectively utilize the periodic characteristics of time series data for optimization. Moreover, when inserting data, existing indexing methods need to frequently adjust the node structure or recalculate the hash, resulting in higher mobile overhead. When faced with massive amounts of data, this mobile overhead not only affects the insertion efficiency, but also causes the overall performance of the system to degrade. In addition, existing methods lack the prediction and optimization of data attribute distribution, resulting in redundant operations during the insertion process. To this end, the present application provides a data index construction scheme that can solve the problems of slowing down the speed of data query using data indexes and increasing mobile overhead when inserting data.
[0053] See also Figure 1 As shown, an embodiment of the present invention discloses a data index construction method, which may include:
[0054] Step S11: Determine the time granularity setting rule corresponding to the target time series data.
[0055] It is understandable that time series data usually has a strong periodicity, and financial transaction time series data has financial transaction peaks and troughs within a day. Therefore, the periodicity of time series data can be used to group time series data according to larger time granularity, and further divide it into smaller time granularity according to needs. In order to facilitate data index construction and subsequent data query, it is necessary to determine the most appropriate time granularity. Large time granularity ensures that the grouped data is relevant and complete, while small time granularity makes the amount of data in each group moderate, which is convenient for subsequent processing and data index construction. In a specific implementation, see Figure 2 As shown in the diagram of data grouping based on time granularity, one day can be used as the large time granularity and one hour as the small time granularity. Then every 24 hours is a cycle, and the data of each hour is a group. Each group of time series data is stored in chronological order. That is, the time series data is divided into larger time granularity and smaller time granularity. and smaller time granularity .set up =24 hours, = 1 hour, each set of data Included in time period After the time granularity is set and grouped for the time series data, each group of time series data is stored in the corresponding time period, and each group is arranged in chronological order. The grouping and time granularity setting based on the periodic characteristics of the time series data makes the distribution of the time series data between large time granularity and small time granularity more reasonable, and improves the efficiency of data processing.
[0056] Step S12: setting the current time granularity corresponding to the current time point according to the time granularity setting rule, and determining the historical time granularity corresponding to the current time granularity.
[0057] In this embodiment, setting the current time granularity corresponding to the current time point according to the time granularity setting rule and determining the historical time granularity corresponding to the current time granularity may include: setting the first time granularity corresponding to the current time point and the second time granularity within the first time granularity according to the time granularity setting rule, and using the second time granularity as the current time granularity; determining the third time granularity corresponding to the first time granularity and before the current time point, and determining the fourth time granularity within the third time granularity and corresponding to the current time granularity, and using the fourth time granularity as the historical time granularity. =24 hours, =1 hour is the time granularity setting rule. Assume that the current time point is The next time point is , then the current time granularity is , the historical time granularity corresponding to the current time granularity is .
[0058] Step S13, using the historical data attributes of the historical time series data within the historical time granularity to predict the first target data attribute distribution of the target time series data within the current time granularity, and constructing the first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index.
[0059] In this embodiment, using the historical data attributes of the historical time series data within the historical time granularity to predict the first target data attribute distribution of the target time series data within the current time granularity may include: determining the first cumulative probability corresponding to the historical data attribute of the historical time series data within the historical time granularity according to a preset cumulative probability function; performing learning index fitting based on the first cumulative probability to obtain a historical position function corresponding to the historical time series data, and predicting the first target data attribute distribution of the target time series data within the current time granularity based on the historical position function; the historical position function is used to characterize the mapping relationship between the historical time series data and the historical index corresponding to the historical time series data. Specifically, if the current time granularity is , the historical time granularity corresponding to the current time granularity is , then we can determine the value of Historical time series data within the time granularity, i.e. The first cumulative probability corresponding to the data attribute is based on historical time series data The first cumulative probability corresponding to the data attribute is learned and indexed to obtain the historical time series data The corresponding historical position function is , and based on the historical position function Predict the current time granularity Target time series data within The first target data attribute distribution. When fitting the first cumulative probability with a learning index, it is necessary to select a suitable model for fitting according to the type and distribution characteristics of the first cumulative probability data. For example, for continuous data, a regression model can be used for fitting; for discrete data, a classification model can be used for fitting. Before fitting, the first cumulative probability data needs to be preprocessed, including data cleaning, data transformation, etc., to ensure the accuracy and consistency of the data. Among them, The corresponding preset cumulative probability function may specifically be:
[0060] ;
[0061] In the formula, Represents historical time series data, Represents the attributes of historical time series data, Represented as data in historical time series data, Represents the attribute value of the data.
[0062] It should be noted that predicting the first target data attribute distribution of the target time series data within the current time granularity based on the historical position function may include: increasing the slope of the historical position function to obtain the first target position function, and predicting the first target data attribute distribution of the target time series data within the current time granularity based on the first target position function; correspondingly, constructing the first initial index of the target time series data based on the first target data attribute distribution may include: constructing the first initial index of the target time series data based on the first target data attribute distribution and a preset gap array. For details, see Figure 3 As shown, if the current time granularity is , the historical time granularity corresponding to the current time granularity is , historical time series data The position function is , then based on the historical position function Predict the current time granularity Target time series data within The first target data attribute distribution is obtained by adding the historical position function The slope , let the target time series data The first target position function is , and based on the first target position function Predict the first target data attribute distribution of the target time series data within the current time granularity. Reserve sufficient gaps in the index through the gap array, and build the first initial index of the target time series data based on the first target data attribute distribution. The use of the gap array when building the first initial index can reduce data movement during data insertion, ensuring that the data insertion operation is efficient and fast.
[0063] It is understandable that after the first initial index of the target time series data is constructed, the target time series data can be inserted based on the first initial index. As the target time series data is inserted, the first initial index of the target time series data is continuously updated. After the target time series data is inserted, the first initial index is updated and the second initial index is obtained.
[0064] Step S14: after the current time point exceeds the time period corresponding to the current time granularity, determine the second target data attribute distribution of the target time series data, and update the second initial index of the target time series data based on the second target data attribute distribution to obtain the target index of the target time series data.
[0065] In this embodiment, as time goes by, the time series data is also updated continuously, so the data index needs to be updated at any time to maintain efficiency and accuracy. , at this time arrive The target time series data between has been stable and is no longer updated. At this time, the target index can be established based on the attributes of the time series data in this time period. Since the time series data in this time period is no longer updated, the data index does not need to use a gap array, thus saving space.
[0066] It should be noted that determining the second target data attribute distribution of the target time series data, and updating the second initial index of the target time series data based on the second target data attribute distribution to obtain the target index of the target time series data, may include: determining the second cumulative probability corresponding to the data attribute value of the target time series data corresponding to the second initial index according to the preset cumulative probability function; performing learning index fitting based on the second cumulative probability to obtain the second target position function corresponding to the target time series data corresponding to the second initial index, and determining the corresponding second target data attribute distribution based on the second target position function; updating the second initial index of the target time series data based on the second target data attribute distribution to obtain the target index of the target time series data. For details, see Figure 4 As shown, if the current time point is determined to be ,but arrive The data between them has stabilized and is no longer updated, so it can be determined based on the preset cumulative probability function The data of the target time series data within the time granularity, that is, The second cumulative probability corresponding to the data attribute is based on the target time series data The second cumulative probability corresponding to the data attribute is learned and indexed to obtain the target time series data The corresponding second target position function is then determined based on the second target position function; and the second initial index of the target time series data is updated based on the second target data attribute distribution, that is, the gap in the second initial index is deleted to obtain the target index of the target time series data. The corresponding preset cumulative probability function may specifically be:
[0067] ;
[0068] In the formula, represents the target time series data, Represents the attributes of the target time series data, Represented as the data in the target time series data, Represents the attribute value of the data.
[0069] It is understandable that when determining the target index of the target time series data, the data attribute distribution of the historical time series data can be used to pre-build a time granularity of between The data index is allocated and the corresponding space is allocated.
[0070] As can be seen from the above, in this embodiment, the time granularity setting rule corresponding to the target time series data is first determined; then the current time granularity corresponding to the current time point is set according to the time granularity setting rule, and the historical time granularity corresponding to the current time granularity is determined; then the data attribute of the historical time series data within the historical time granularity is used to predict the first target data attribute of the target time series data within the current time granularity, and the initial index of the target time series data is constructed based on the first target data attribute; finally, after the current time point exceeds the time period corresponding to the current time granularity, the second target data attribute of the target time series data is determined, and the initial index of the target time series data is updated based on the second target data attribute to obtain the target index of the target time series data. As can be seen from the above, this embodiment utilizes the periodic characteristics of time series data, and sets appropriate time granularity to group data, ensures the relevance and integrity of data, and can use both large and small time granularities to optimize index construction and query efficiency; then, based on the data attributes of the historical target time series data, construct the initial index of the target time series data, and pre-allocate the index space, so that the mobile overhead when inserting the target time series data can be reduced, and the data processing efficiency can be significantly improved; then, the initial index of the target time series data is updated using the data attributes of the target time series data in a stable time period to obtain the target index, so that the data index can be updated in real time to maintain high efficiency and accuracy; when querying data, the required data can be quickly and accurately found based on the time granularity, attributes and target index. In this way, this embodiment can solve the problem of slowing down the speed of data query and increasing the mobile overhead when inserting data by using data index through the above process.
[0071] Through the above embodiment, the historical data attributes of the historical time series data are used to predict the data attribute distribution of the target time series data and to construct an initial index based on the data attribute distribution of the target time series data. Then, the initial index is updated to obtain the target index of the target time series data. Next, in order to query the data in the target time series data based on the target index, see Figure 5 As shown, the embodiment of the present invention further discloses a data index construction method, which may include:
[0072] Step S21: Determine the time period to be queried and the range of attribute values to be queried of the preset query condition.
[0073] In this embodiment, in order to query the data in the target time series data based on the target index, it is first necessary to determine the query time period and the query attribute value range of the preset query condition, such as querying a specific time period. The attribute value is within a specific range All data within.
[0074] Step S22, determining a corresponding target time granularity according to the time period to be queried, and determining a time series data group corresponding to the target time granularity; wherein each of the time series data groups includes the target time series data within the corresponding target time granularity.
[0075] In this embodiment, the time stamp is used for preliminary screening to quickly lock the time period to be queried. In specific implementation, the constructed time granularity can be used to quickly locate the and Corresponding target time granularity , and determine the target time granularity The corresponding time series data set and , the data obtained after preliminary screening are .
[0076] Step S23: traverse each of the time series data groups based on the target index and the attribute value range to be queried to obtain target query data.
[0077] It should be noted that after locking the time series data group, the target index of the target time series data is used to further accurately search for the required data. Corresponding data First, find the attribute value by the target index in The time series data in the query is added to the query results, and then the next set of data is processed, until Corresponding data The final target query data is .
[0078] Step S24: store the target query data corresponding to the minimum attribute value within the range of attribute values to be queried in each of the time series data groups into a preset priority queue; merge and sort the target query data using the preset priority queue according to the attribute value of the target query data to obtain the sorted target query data.
[0079] Understandably, see Figure 6 As shown, after determining the target query data, if the target query data needs to be sorted, a similar idea to merge sort can be adopted, using a preset priority queue to maintain the minimum value in each time series data group, obtaining the minimum value in the preset priority queue each time, and inserting the next minimum value in the corresponding time series data group into the priority queue. In a specific implementation, if you want to query data from 1:00 to 4:00 on September 20, 2024, with data attribute values between 10 and 15 and sort the target query data, you can determine that the target time granularity corresponding to the time period to be queried is , , , the range of attribute values to be queried is , then the time granularity can be determined as The data attributes in the time series data group are: 3, 4, 10, 12, 15, 17, 18, 19, and the time granularity is The data attributes in the time series data group are: 1, 6, 7, 11, 14, 15, 17, 20, and the time granularity is The data attributes in the time series data group are: 2, 3, 9, 12, 13, 14, 16, 20. First, we can select the time granularity as The data attribute value in the time series data group is 12 and the time granularity is The data attribute value in the time series data group is 11 and the time granularity is The data attribute value 12 in the time series data group enters the preset priority queue, and the query result of the data attribute is: 11, 12. Then, the next minimum value in each time series data group is continuously inserted into the priority queue, and the final query result of the data attribute is: 10, 11, 12, 12, 13, 14, 14, 15, 15. In this way, the preset priority queue can be used to merge and sort the target query data according to the attribute value of the target query data to obtain the sorted target query data.
[0080] In this embodiment, see Figure 7 As shown, the specific steps for data query based on data index are: first determine the time period and query granularity of the data, then pre-build a learning index based on the data of the same small time granularity in the previous time period, that is, the initial index, and then rebuild the learning index to obtain the target index after the data in a small time granularity is no longer updated. Finally, the data with attribute values within a specified range within a certain time period can be queried based on the target index.
[0081] As can be seen from the above, this embodiment first determines the time period to be queried and the range of attribute values to be queried of the preset query condition; then determines the corresponding target time granularity according to the time period to be queried, and determines the time series data group corresponding to the target time granularity; wherein each time series data group includes the target time series data within the corresponding target time granularity; then traverses each time series data group based on the target index and the range of attribute values to be queried to obtain the target query data; finally, the target query data corresponding to the minimum attribute value within the range of attribute values to be queried in each time series data group can be stored in the preset priority queue; the target query data is merged and sorted using the preset priority queue according to the attribute value of the target query data to obtain the sorted target query data. As can be seen from the above, the time period to be queried and the attribute value to be queried can be determined according to the preset query condition, and the corresponding time series data group can be determined based on the time period to be queried, and then the target query data in the time series data group can be queried based on the attribute value to be queried, and the target query data can also be sorted according to the attribute value of the target query data based on the preset priority queue and merge sort. In this way, this embodiment combines time granularity, attributes and target indexes to perform efficient data query, quickly locks the time series data group through preliminary screening, and accurately finds the required data. When sorting is required, the priority queue is used for merge sorting. In this way, this embodiment can solve the problem of slowing down the data query by using data index through the above process.
[0082] Accordingly, see Figure 8 As shown, the embodiment of the present application also provides a data index construction device, which may include:
[0083] A rule determination module 11 is used to determine a time granularity setting rule corresponding to the target time series data;
[0084] A time granularity determination module 12 is used to set the current time granularity corresponding to the current time point according to the time granularity setting rule, and determine the historical time granularity corresponding to the current time granularity;
[0085] An index building module 13 is used to predict a first target data attribute distribution of the target time series data within the current time granularity by using the historical data attributes of the historical time series data within the historical time granularity, and to build a first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index;
[0086] The index update module 14 is used to determine the second target data attribute distribution of the target time series data after the current time point exceeds the time period corresponding to the current time granularity, and update the second initial index of the target time series data based on the second target data attribute distribution to obtain the target index of the target time series data.
[0087] As can be seen from the above, in this embodiment, the time granularity setting rule corresponding to the target time series data is first determined; then the current time granularity corresponding to the current time point is set according to the time granularity setting rule, and the historical time granularity corresponding to the current time granularity is determined; then the data attribute of the historical time series data within the historical time granularity is used to predict the first target data attribute of the target time series data within the current time granularity, and the initial index of the target time series data is constructed based on the first target data attribute; finally, after the current time point exceeds the time period corresponding to the current time granularity, the second target data attribute of the target time series data is determined, and the initial index of the target time series data is updated based on the second target data attribute to obtain the target index of the target time series data. As can be seen from the above, this embodiment sets a time granularity setting rule for time series data, and groups the target time series data by setting an appropriate time granularity, so that the efficiency of data processing can be improved; then, based on the data attributes of the historical target time series data, the initial index of the target time series data is constructed, and the index space is pre-allocated, so that the mobile overhead when the target time series data is inserted can be reduced, and the data processing efficiency can be significantly improved; then, the initial index of the target time series data is updated using the data attributes of the target time series data in the stable time period to obtain the target index, and the required data can be quickly and accurately found based on the time granularity, attributes and target index when querying data. In this way, this embodiment can solve the problem of slowing down the speed of data query and increasing the mobile overhead when inserting data by using the data index through the above process.
[0088] In some specific implementations, the time granularity determination module 12 includes:
[0089] A time granularity determination unit is used to set a first time granularity corresponding to the current time point and a second time granularity within the first time granularity according to the time granularity setting rule, and use the second time granularity as the current time granularity; determine a third time granularity corresponding to the first time granularity and before the current time point, and determine a fourth time granularity within the third time granularity and corresponding to the current time granularity, so as to use the fourth time granularity as the historical time granularity.
[0090] In some specific implementations, the index building module 13 includes:
[0091] A first cumulative probability determination unit, configured to determine a first cumulative probability corresponding to the historical data attribute of the historical time series data within the historical time granularity according to a preset cumulative probability function;
[0092] A function determination unit is used to perform learning index fitting based on the first cumulative probability to obtain a historical position function corresponding to the historical time series data, and predict the first target data attribute distribution of the target time series data within the current time granularity based on the historical position function; the historical position function is used to characterize the mapping relationship between the historical time series data and the historical index corresponding to the historical time series data.
[0093] In some specific implementations, the index building module 13 includes:
[0094] The attribute distribution prediction unit is used to increase the slope of the historical position function to obtain a first target position function, and predict the first target data attribute distribution of the target time series data within the current time granularity based on the first target position function.
[0095] Accordingly, the index building module 13 includes:
[0096] An initial index building unit is used to build the first initial index of the target time series data based on the first target data attribute distribution and a preset gap array.
[0097] In some specific implementations, the index updating module 14 includes:
[0098] A second cumulative probability determination unit, configured to determine, according to the preset cumulative probability function, a second cumulative probability corresponding to the data attribute value of the target time series data corresponding to the second initial index;
[0099] an attribute distribution determining unit, configured to perform learning index fitting based on the second cumulative probability to obtain a second target position function corresponding to the target time series data corresponding to the second initial index, and determine the corresponding attribute distribution of the second target data based on the second target position function;
[0100] A target index determination unit is used to update the second initial index of the target time series data based on the second target data attribute distribution to obtain a target index of the target time series data.
[0101] In some specific implementations, the data index construction device further includes:
[0102] A condition determination module, used to determine the query time period and the attribute value range of the preset query condition;
[0103] A data group determination module, used to determine the corresponding target time granularity according to the time period to be queried, and determine the time series data group corresponding to the target time granularity; wherein each of the time series data groups includes the target time series data within the corresponding target time granularity;
[0104] The data query module is used to traverse each of the time series data groups based on the target index and the range of attribute values to be queried to obtain target query data.
[0105] In some specific implementations, the data index construction device further includes:
[0106] A data storage module, used for storing the target query data corresponding to the minimum attribute value within the to-be-queried attribute value range in each of the time series data groups into a preset priority queue;
[0107] The data sorting module is used to merge and sort the target query data using the preset priority queue according to the attribute value of the target query data to obtain the sorted target query data.
[0108] Furthermore, the present application also discloses an electronic device. Fig. 9 It is a structural diagram of an electronic device 20 shown according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the data index construction method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment can specifically be an electronic computer.
[0109] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0110] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0111] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data index construction method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0112] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed data index construction method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.
[0113] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0114] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0115] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0116] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0117] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data index construction method, characterized in that: include: Determine the time granularity setting rules corresponding to the target time series data; Setting the current time granularity corresponding to the current time point according to the time granularity setting rule, and determining the historical time granularity corresponding to the current time granularity; Predicting a first target data attribute distribution of the target time series data within the current time granularity by using the historical data attributes of the historical time series data within the historical time granularity, and constructing a first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index; After the current time point exceeds the time period corresponding to the current time granularity, a second target data attribute distribution of the target time series data is determined, and the second initial index of the target time series data is updated based on the second target data attribute distribution to obtain a target index of the target time series data.
2. The data index construction method according to claim 1, characterized in that: The step of setting the current time granularity corresponding to the current time point according to the time granularity setting rule, and determining the historical time granularity corresponding to the current time granularity, includes: According to the time granularity setting rule, a first time granularity corresponding to the current time point and a second time granularity within the first time granularity are set, and the second time granularity is used as the current time granularity; A third time granularity corresponding to the first time granularity and before the current time point is determined, and a fourth time granularity within the third time granularity and corresponding to the current time granularity is determined, so as to use the fourth time granularity as the historical time granularity.
3. The data index construction method according to claim 2, characterized in that: The method of using the historical data attributes of the historical time series data within the historical time granularity to predict the first target data attribute distribution of the target time series data within the current time granularity includes: Determine, according to a preset cumulative probability function, a first cumulative probability corresponding to the historical data attribute of the historical time series data within the historical time granularity; Based on the first cumulative probability, learning index fitting is performed to obtain a historical position function corresponding to the historical time series data, and based on the historical position function, the first target data attribute distribution of the target time series data within the current time granularity is predicted; the historical position function is used to characterize the mapping relationship between the historical time series data and the historical index corresponding to the historical time series data.
4. The data index construction method according to claim 3, characterized in that: The predicting the first target data attribute distribution of the target time series data within the current time granularity based on the historical position function includes: Increasing the slope of the historical position function to obtain a first target position function, and predicting the first target data attribute distribution of the target time series data within the current time granularity based on the first target position function; Accordingly, constructing a first initial index of the target time series data based on the first target data attribute distribution includes: The first initial index of the target time series data is constructed based on the first target data attribute distribution and a preset gap array.
5. The data index construction method according to claim 4, characterized in that: The determining of a second target data attribute distribution of the target time series data, and updating the second initial index of the target time series data based on the second target data attribute distribution to obtain a target index of the target time series data, includes: Determine, according to the preset cumulative probability function, a second cumulative probability corresponding to the data attribute value of the target time series data corresponding to the second initial index; Performing learning index fitting based on the second cumulative probability to obtain a second target position function corresponding to the target time series data corresponding to the second initial index, and determining the corresponding second target data attribute distribution based on the second target position function; The second initial index of the target time series data is updated based on the second target data attribute distribution to obtain a target index of the target time series data.
6. The data index construction method according to any one of claims 1 to 5, characterized in that: Also includes: Determine the query time period and the query attribute value range of the preset query condition; Determine the corresponding target time granularity according to the time period to be queried, and determine the time series data group corresponding to the target time granularity; wherein each of the time series data groups includes the target time series data within the corresponding target time granularity; The target query data is obtained by traversing each of the time series data groups based on the target index and the attribute value range to be queried.
7. The data index construction method according to claim 6, characterized in that: After traversing each of the time series data groups based on the target index and the range of the attribute value to be queried to obtain the target query data, the method further includes: The target query data corresponding to the minimum attribute value within the to-be-queried attribute value range in each of the time series data groups is stored in a preset priority queue; The target query data is merged and sorted using the preset priority queue according to the attribute value of the target query data to obtain the sorted target query data.
8. A data index construction device, characterized in that: include: A rule determination module is used to determine the time granularity setting rule corresponding to the target time series data; A time granularity determination module, used to set the current time granularity corresponding to the current time point according to the time granularity setting rule, and determine the historical time granularity corresponding to the current time granularity; An index building module, used to predict a first target data attribute distribution of the target time series data within the current time granularity by using the historical data attributes of the historical time series data within the historical time granularity, and to build a first initial index of the target time series data based on the first target data attribute distribution, so as to update the first initial index according to the data attribute value of the target time series data within the current time granularity to obtain a second initial index; An index updating module is used to determine a second target data attribute distribution of the target time series data after the current time point exceeds the time period corresponding to the current time granularity, and to update the second initial index of the target time series data based on the second target data attribute distribution to obtain a target index of the target time series data.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data index construction method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the data index construction method according to any one of claims 1 to 7 is implemented.