A data retrieval method and device based on a timing intermediate state, equipment, medium and product

Abstract

The application discloses a data retrieval method and device based on a time sequence intermediate state, equipment, medium and product, relates to the field of data analysis and calculation, and comprises the following steps: extracting a retrieval condition from a data retrieval task; matching an intermediate state index currently established in a database with the retrieval condition; the intermediate state index is an index established based on the calculation of a preset operator on time sequence data in a target group, the target group is a group obtained by dividing the time sequence data in the database, each intermediate state index comprises time slices of the target group and corresponding operator calculation results; if there is a target intermediate state index matching the retrieval condition in the database, corresponding retrieval results are returned based on the target intermediate state index. The application establishes an intermediate state index for time sequence data in a database, and improves data retrieval and calculation efficiency.

Description

Technical Field

[0001] This invention relates to the field of data analysis and computing, and in particular to a data retrieval method, apparatus, device, medium, and product based on time-series intermediate states. Background Technology

[0002] In the era of big data, time-series data in fields such as the Internet, the Internet of Things (IoT), and finance is experiencing explosive growth. Real-time applications such as anti-telecom fraud, financial risk control, and IoT status monitoring place higher demands on the efficiency and real-time nature of data retrieval. Retrieval tasks in these scenarios typically involve aggregation calculations, such as calculating the total amount or number of transactions of a certain type within a specific time range, requiring the rapid extraction of effective information from massive amounts of time-series data.

[0003] In existing technologies, common relational or distributed databases such as PostgreSQL (object-relational database) and HBase (distributed database) generally suffer from performance bottlenecks when handling retrieval requests containing aggregation operators (such as summation, counting, variance, etc.). Traditional solutions typically require a full table scan of all detailed data involved in the retrieval and real-time calculation of aggregation results during the retrieval process, resulting in high retrieval latency and making it difficult to meet the business requirements of high concurrency and low latency. To alleviate this problem, the industry often uses indexing to reduce the amount of detailed data to be processed, but traditional indexing solutions still have significant limitations: on the one hand, index creation must strictly match the retrieval logic, otherwise index failure is likely to occur, and specific design principles must be followed, requiring a high level of professional skills from relevant personnel; on the other hand, traditional index construction lacks flexibility, with fixed time sharding granularity, making it unable to adapt to retrieval needs in different scenarios, and offering limited performance improvement when dealing with large-scale time-series streaming data, not only failing to meet the needs of efficient retrieval but also consuming a large amount of server resources and increasing hardware and software deployment costs.

[0004] In summary, how to improve data retrieval and computation efficiency by creating appropriate indexes is a problem that needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a data retrieval method, apparatus, device, medium, and product based on time-series intermediate states, which can improve data retrieval and computation efficiency by establishing appropriate indexes. The specific solution is as follows:

[0006] Firstly, this application discloses a data retrieval method based on temporal intermediate states, including:

[0007] Extract search criteria from data retrieval tasks;

[0008] The intermediate index currently established in the database is matched with the search conditions; the intermediate index is an index established based on the calculation of time series data in the target group by a preset operator, the target group is a group obtained by dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result;

[0009] If a target intermediate state index matching the search criteria exists in the database, the corresponding search result is returned based on the target intermediate state index.

[0010] In the first aspect mentioned above, this application pre-groups the time-series data in the database and calculates the time-series data within each group based on preset operators to establish an intermediate-state index. Each intermediate-state index includes a time slice of the corresponding group and the corresponding operator calculation result. When a data retrieval task is obtained, the retrieval conditions extracted from the data retrieval task are first matched with the established intermediate-state index. If a target intermediate-state index is successfully matched, the corresponding retrieval result can be directly returned based on the target intermediate-state index. That is, since this application has pre-established the intermediate-state index, the operator calculation result can be directly reused when a match is successful, without having to repeatedly scan or calculate the original detailed data, significantly shortening the retrieval response time and improving the efficiency of data retrieval and calculation. It can be seen that this application, for this type of data with time-series characteristics in the database, pre-builds an intermediate-state index for the data through the time dimension, effectively solving the problem that the current calculation needs to traverse detailed data, and improving the calculation speed.

[0011] In one possible implementation of the first aspect, the search criteria include a search time range and a task operator type;

[0012] Accordingly, matching the currently established intermediate indexes in the database with the search conditions includes:

[0013] If any intermediate index in the database currently exists has a time shard that covers a time range less than or equal to the retrieval time range, and the calculation result of the operator corresponding to any intermediate index matches the task operator type, then that intermediate index will be used as the target intermediate index that matches the retrieval conditions.

[0014] Accordingly, returning the corresponding search results based on the target intermediate index includes:

[0015] The corresponding search results are constructed based on the operator calculation results corresponding to the target intermediate state index, and the search results are returned.

[0016] In this possible implementation, matching is performed from two perspectives: the search time range and the task operator type. When a matching target intermediate state index exists in the database, the operator calculation result of the intermediate state index is directly reused, which can quickly obtain the calculation result and improve the calculation speed. Therefore, it is not necessary to repeat the calculation process on the time series data corresponding to the intermediate state index.

[0017] In one possible implementation of the first aspect, the data retrieval method based on time-series intermediate states further includes:

[0018] If none of the intermediate indexes currently established in the database have an operator calculation result that matches the task operator type, and at least two intermediate indexes have operator calculation results that have a preset algorithm dependency relationship with the task operator type, then the at least two intermediate indexes are determined as target intermediate indexes that match the search conditions.

[0019] Wherein, the time slices corresponding to the at least two intermediate state indexes are the same, the corresponding operator calculation results are different, and the time range covered by the time slices corresponding to the at least two intermediate state indexes is less than or equal to the retrieval time range;

[0020] Accordingly, returning the corresponding search results based on the target intermediate index includes:

[0021] The calculation results of the operators corresponding to the at least two intermediate state indices are calculated according to the preset algorithm dependency relationship to obtain the target calculation result that matches the task operator type;

[0022] Based on the target calculation results, construct the corresponding search results and return the search results.

[0023] In this possible implementation, when the operator calculation result corresponding to any intermediate index in the database does not match the task operator type, it is also possible to query whether there are at least two types of intermediate indexes in the current database that have a preset algorithm dependency relationship with the task operator type. If they exist, the operator calculation results corresponding to these two intermediate indexes can be reused and calculated according to the preset algorithm dependency relationship. Compared with full data scanning, this method also reduces the amount of calculation and improves the calculation speed.

[0024] In one possible implementation of the first aspect, the data retrieval method based on time-series intermediate states further includes:

[0025] If none of the intermediate indexes currently established in the database cover a time range less than or equal to the retrieval time range, then it is determined that there is no target intermediate index in the database that matches the retrieval conditions.

[0026] Alternatively, if any intermediate index in the database currently has a time shard covering a time range less than or equal to the retrieval time range, and no intermediate index has an operator calculation result that matches the task operator type, and no two intermediate indexes have operator calculation results that have a preset algorithm dependency relationship with the task operator type, then it is determined that there is no target intermediate index in the database that matches the retrieval conditions.

[0027] In this possible implementation, two scenarios in which the intermediate index cannot be matched are specifically given. Through multi-level condition judgment, the intermediate index cannot be reused under which circumstances.

[0028] In one possible implementation of the first aspect, returning the corresponding search results based on the target intermediate index includes:

[0029] If no target intermediate state index matching the search criteria exists in the database, then first time-series data that meets the search time range is queried from the database.

[0030] The task operator type is used to calculate the first time series data to obtain a first calculation result, and the retrieval result constructed based on the first calculation result is returned.

[0031] This possible implementation further provides how to calculate detailed data using traditional methods when the established intermediate index cannot be reused, thus ensuring the integrity of the solution.

[0032] In one possible implementation of the first aspect, the search criteria further include a business dimension field for characterizing the business type of the data; the target group is a group obtained by dividing the time-series data in the database according to different business types; each intermediate index further includes the target business type of the target group;

[0033] Accordingly, matching the currently established intermediate indexes in the database with the search conditions includes:

[0034] Determine whether any of the currently established intermediate indexes in the database matches the target business type of the business dimension field in the search criteria.

[0035] If it exists, the initial intermediate index that matches the business dimension field in the search conditions is selected, and the time slicing and operator calculation results corresponding to the initial intermediate index are matched with the search time range and task operator type in the search conditions respectively to determine whether there is a target intermediate index that matches the search conditions in the initial intermediate index.

[0036] If no intermediate index matches the business dimension field in the search criteria, it is determined that no target intermediate index matches the search criteria in the database.

[0037] In one possible implementation, to enhance the analytical capabilities of the index, when creating the intermediate index, corresponding indexes are created based on time-series data of different business types. Correspondingly, the search criteria can also include targeted business dimension fields, thereby filtering irrelevant index information based on the business dimension fields and the search time range. In this way, retrieval can be performed only on data of a specific business type.

[0038] In one possible implementation of the first aspect, each intermediate index further includes the start time of the time-series data within the target group;

[0039] Accordingly, the time range covered by the time shard corresponding to any intermediate index is determined, including:

[0040] The time range covered by any intermediate index is determined based on the start time and time slice corresponding to any intermediate index.

[0041] In this possible implementation, the intermediate state index includes not only time slices and corresponding operator calculation results, but also the start time of time-series data within the target group, thereby enabling accurate calculation of the time range covered by the intermediate state index.

[0042] In one possible implementation of the first aspect, the data retrieval method based on time-series intermediate states further includes:

[0043] If the sum of the time ranges covered by each of the target intermediate indexes is less than the retrieval time range, then the remaining time ranges not covered by the target intermediate indexes are determined.

[0044] The second time-series data that matches the remaining time range is queried from the preset index structure tree, and the second time-series data is calculated using the task operator type to obtain a second calculation result; wherein, the index structure tree is constructed based on all time-series data and intermediate state indexes in the database;

[0045] Accordingly, returning the corresponding search results based on the target intermediate index includes:

[0046] The target intermediate state index and the second calculation result are summarized to construct the corresponding search result, and the search result is returned.

[0047] In this possible implementation, considering that the time range covered by each intermediate index is not completely aligned with the retrieval time range, this application achieves accurate support for queries within any retrieval time range by dynamically reusing the index and supplementing the data within the remaining time range as needed.

[0048] In one possible implementation of the first aspect, each leaf node in the index structure tree corresponds to a time-series data, and leaf nodes with adjacent time sequences are connected by pointers.

[0049] In this possible implementation, by constructing an index structure tree and connecting leaf nodes that are adjacent in time sequence with pointers, the query efficiency of data within the remaining time range is improved.

[0050] In one possible implementation of the first aspect, the process of dividing the time-series data in the database into groups includes:

[0051] The time-series data in the database is divided into several groups according to fixed time segments;

[0052] Alternatively, the historical time-series data stored in the database can be analyzed to determine the data density corresponding to different time periods, so as to establish a correspondence between time periods and time slice sizes based on the data density; wherein the data density and the time slice size are negatively correlated.

[0053] Determine the target time period to which the time series data to be divided belongs, and divide the time series data to be divided into several groups using the target time segment size corresponding to the target time period.

[0054] In this possible implementation, when grouping time-series segments, a fixed-time-segmentation mechanism or an adaptive-time-segmentation mechanism can be used to ensure the flexibility of grouping, thus making it applicable to different scenarios.

[0055] In one possible implementation of the first aspect, the data retrieval method based on time-series intermediate states further includes:

[0056] The matching index aggregation precision is determined based on the cumulative storage time of the time-series data in the database;

[0057] Based on the index aggregation precision, a corresponding number of intermediate state indices are aggregated to obtain aggregated intermediate state indices; wherein, the time slice corresponding to the aggregated intermediate state index is the sum of the time slices corresponding to the corresponding number of intermediate state indices, and the operator calculation result corresponding to the aggregated intermediate state index is the sum of the operator calculation results corresponding to the corresponding number of intermediate state indices.

[0058] In this possible implementation, as the storage time of time-series data accumulates, the data becomes less popular, so multiple intermediate indexes can be aggregated to free up storage space.

[0059] Secondly, this application discloses a data retrieval device based on a time-series intermediate state, comprising:

[0060] The condition extraction module is used to extract search conditions from data retrieval tasks;

[0061] The judgment module is used to match the currently established intermediate indexes in the database with the search conditions; the intermediate index is an index established based on the calculation of time series data in the target group by a preset operator, the target group is a group obtained by dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result;

[0062] The result return module is used to return the corresponding search result based on the target intermediate index if a target intermediate index that matches the search conditions exists in the database.

[0063] Thirdly, this application discloses an electronic device, including:

[0064] Memory, used to store computer programs;

[0065] A processor is configured to execute the computer program to implement the steps of the aforementioned disclosed data retrieval method based on time-series intermediate states.

[0066] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the steps of the aforementioned disclosed data retrieval method based on time-series intermediate states.

[0067] Fifthly, the present invention discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned disclosed data retrieval method based on time-series intermediate states. Attached Figure Description

[0068] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0069] Figure 1 This is a schematic diagram of a system architecture disclosed in this application;

[0070] Figure 2 This is a flowchart of a data retrieval method based on temporal intermediate states disclosed in this application;

[0071] Figure 3 This is a schematic diagram of the data structure of a time-series data set disclosed in this application;

[0072] Figure 4 This is a schematic diagram of a data retrieval process disclosed in this application;

[0073] Figure 5 This is a schematic diagram of an index structure tree and intermediate index disclosed in this application;

[0074] Figure 6 This is a flowchart illustrating the merging process of detailed data and intermediate indexes as disclosed in this application;

[0075] Figure 7 This is a schematic diagram of the structure of a data retrieval device based on a temporal intermediate state disclosed in this application;

[0076] Figure 8 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0078] This application discloses a data retrieval method, apparatus, device, medium, and product based on temporal intermediate states, which can improve data retrieval and computation efficiency by establishing appropriate indexes.

[0079] First, we will explain some of the terms and concepts involved in the data retrieval method based on temporal intermediate states provided in the embodiments of this application.

[0080] Temporal intermediate states refer to data that has a time sequence. When an intelligent decision is needed, it provides a real-time, dynamic reorganization and incremental concatenation. Different time intervals generate different data metrics, resulting in very fast, flexible, and intelligent overall performance. Temporal intermediate states aim to solve the real-time processing needs in high-concurrency data scenarios. This technology achieves millisecond-level computation through dynamic reorganization and incremental concatenation of data.

[0081] Time slicing refers to dividing continuous time series data (such as millisecond-level timestamps) into several discrete time periods of fixed or variable length, each time period being called a slice.

[0082] Intermediate state: This is another key concept relative to the original state and the final result state. From the perspective of the entire data processing workflow, it is a semi-processed state formed after the original time-series data has undergone preliminary aggregation and calculation. It retains the core information required for calculation but does not directly generate the final retrieval result. It serves as a bridge connecting the original data and the final result. The original state refers to the unprocessed time-series data in the database, such as the timestamp and amount of each transaction, or the monitoring values ​​generated by sensors per second; this is the most basic unprocessed data. The final result state refers to the answer generated for a specific data retrieval task, such as a user's total transaction amount in the past 5 minutes or the highest temperature of a device in one hour; this is the processed result. The intermediate state, on the other hand, is the state obtained after preliminary calculation of the original data according to preset rules, such as time slicing rules and preset operators. It is not the original data of each transaction, nor is it the final result for a specific retrieval, but it can be directly used as a semi-finished product for calculating the final result.

[0083] The core characteristics of intermediate states include pre-computation and partial aggregation. Pre-computation means that it is generated before the data retrieval task occurs, rather than being calculated temporarily during retrieval, so it can be directly reused. Partial aggregation means that it is the result of aggregating the original data for a certain local time range, rather than the final aggregation of the entire data, so it can be flexibly combined, such as merging multiple 10-minute intermediate states into a 1-hour result.

[0084] Figure 1 This is a schematic diagram of a system architecture applicable to an embodiment of this application, such as... Figure 1 As shown, it mainly includes client 01, server 02, and database 03.

[0085] Client 01 serves as the entry point for applications to initiate data retrieval tasks. Clients can be further divided into real-time retrieval clients and non-real-time retrieval clients, and there can be multiple clients 01. Real-time retrieval clients include financial risk control terminals and risk control system clients, while non-real-time retrieval clients include report analysis platforms and data analysis platforms. The client primarily receives user-input retrieval requests, such as "query user A's total transaction amount in the last 10 minutes," encapsulates the retrieval request into a standardized data retrieval task, and then sends it to server 02. It also receives and displays the retrieval results returned by server 02. Client 01 communicates only unidirectionally with server 02, i.e., sending data retrieval tasks → receiving retrieval results, and does not directly access database 03.

[0086] Server 02 is used to receive data retrieval tasks initiated by client 01. The server is equipped with a retrieval engine, which is mainly responsible for the execution flow of retrieval logic. Specifically, it is responsible for parsing retrieval conditions, matching intermediate indexes from database 03, constructing retrieval results, and returning retrieval results to client 01.

[0087] Database 03 comprises an intermediate index repository and a time-series database, responsible for data persistence. The intermediate index repository can utilize a structured database supporting B-tree indexes. The intermediate indexes are created by pre-grouping the time-series data in the time-series database according to time-sharding rules and then performing calculations on the time-series data within each group based on preset operators. Each intermediate index includes the time shard of the group and the corresponding operator calculation result. The time-series database stores the raw time-series data and can be either a time-series database or a distributed file system, supporting efficient querying by time range.

[0088] See Figure 2 As shown in the figure, this application discloses a data retrieval method based on a time-series intermediate state, the method comprising:

[0089] Step S11: Extract search criteria from the data retrieval task.

[0090] In this embodiment, when a data retrieval task is obtained, the retrieval conditions are first extracted from the data retrieval task to clarify what data content needs to be retrieved.

[0091] It should be noted that the solution proposed in this application can be applied to both online and offline scenarios. Online scenarios focus on near real-time operations, such as instant analysis and decision-making in transaction processing. Real-time data retrieval tasks have extremely high latency requirements, typically in the millisecond range. For example, a bank needs to conduct real-time risk control monitoring of user credit card transactions. When a user initiates a transaction, the system needs to complete a retrieval judgment within 100 milliseconds to determine whether the user's total transaction amount in the past 5 minutes exceeds 50,000 yuan. If it does, a risk warning is triggered. Offline scenarios mainly involve batch processing operations. Batch processing can be used to calculate the cumulative transaction amount for the day, or to calculate the number of transactions made so far. This type of non-real-time data retrieval task has lower latency requirements but involves extremely large amounts of data. For example, an e-commerce platform needs to execute a task to "statistically calculate the sales revenue of each product category the previous day" every morning to generate sales reports. This task can tolerate a delay of hours.

[0092] Step S12: Match the currently established intermediate indexes in the database with the search conditions; the intermediate indexes are indexes established after calculating the time series data in the target group based on preset operators, the target group is a group obtained after dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result.

[0093] In this embodiment, after the search criteria are extracted, if a traditional approach is adopted, all the required data needs to be fully scanned before calculation, which poses a great challenge to both response efficiency and server resources.

[0094] In this embodiment, the time-series data in the database is pre-grouped, and intermediate-state indexes are established by calculating the time-series data within each group based on preset operators. Each intermediate-state index includes a time slice of the corresponding group and the corresponding operator calculation result. Then, the search criteria extracted from the data retrieval task can be matched with the established intermediate-state indexes to determine whether an intermediate-state index matching the search criteria exists in the database.

[0095] In a specific embodiment, the process of dividing the time-series data in the database into groups includes: dividing the time-series data in the database into several groups according to fixed time slices; or, analyzing the historical time-series data stored in the database to determine the data density corresponding to different time periods, and establishing a correspondence between time periods and time slice sizes based on the data density; wherein the data density and the time slice size are negatively correlated; determining the target time period to which the time-series data to be divided belongs, and dividing the time-series data to be divided into several groups using the target time slice size corresponding to the target time period.

[0096] It is understood that this application uses a preset time sharding rule to group the time-series data in the database; the time sharding rule can be a fixed time sharding mechanism or an adaptive time sharding mechanism. For a set of time-series data, each data point has a time attribute; for example, there is a set of time-series data... Data in a time-series dataset can be grouped according to time-slicing rules to obtain groups: G={ , ... Next, an intermediate state index is built for each group according to the preset operators. The intermediate state index includes the time slice of the group and the corresponding operator calculation result.

[0097] In one specific implementation, the time-series data in the database can be divided into several groups according to fixed-time shards. For example, taking a fixed-time shard of 1 hour as an example, the time-series data within a day can be divided into 24 groups, with each group having a time shard of 1 hour. Furthermore, the granularity of the fixed-time shards can also be in minutes, seconds, or milliseconds. This embodiment does not limit the specific size of the fixed-time shards and supports user-defined settings.

[0098] In another specific implementation, this embodiment also supports adaptive time sharding. The system can dynamically determine an optimal time shard based on the real-time distribution of data in the database to maximize index utilization. Specifically, the shard granularity can be dynamically adjusted according to data density. For example, during peak data periods, such as peak financial transaction periods, the shard granularity is automatically reduced to improve index accuracy; while during periods of sparse data, the shard granularity is automatically expanded to reduce index storage overhead. This adaptive time sharding mechanism can solve the resource waste problem of fixed shards in scenarios with uneven data distribution, and can also improve the utilization rate of the index for different query time ranges during retrieval, effectively reducing the amount of detailed data that needs to be scanned, thereby maximizing index utilization. To achieve this goal, this embodiment can analyze the historical time-series data stored in the database to determine the data density corresponding to different time periods, and establish a correspondence between time periods and time shard sizes based on data density.

[0099] For example, in anti-telecom fraud scenarios, by analyzing time-series data over a period of time, it can be found that a new type of fraudulent method causes a surge in transfers to a certain region between 8:00 AM and 5:00 PM, resulting in a large data volume and high data density. Therefore, the time segmentation granularity during this period can be automatically adjusted to one minute per segment; while at night when data volume is lower, it can be automatically adjusted to one 10-minute segment. This embodiment does not limit the specific size of the time segment. For example, when the data density is high, it can be the aforementioned one minute, or it can be further reduced to half a minute, 10 seconds, etc.; while when the data density is low, it can be the aforementioned 10 minutes, or half an hour, one hour, or even two hours, three hours, etc. However, it should be noted that data density and time segment size are negatively correlated; that is, the higher the data density, the smaller the time segment. Therefore, for time periods with large data volumes, they can be divided into smaller granular time segments. In this way, when it is necessary to divide time series data, the target time period to which the current time series data to be divided belongs is determined, and then the target time slice size corresponding to the target time period is used to divide the time series data to be divided into several groups.

[0100] In addition, this application can also set up an "index advisor" module. This module can continuously analyze users' search history logs, use machine learning to identify the most common search patterns, such as the most frequently queried data types and time ranges, and proactively recommend new and optimized index creation patterns to database administrators (DBAs), thereby transforming index tuning from manual experience-driven to intelligent data-driven.

[0101] It's important to note that once the current time sharding rules are determined, the time-series data in the database will be automatically sharded once it reaches the corresponding time shard. For example, if the current sharding is based on 1-hour intervals, then after 1 hour of accumulated time-series data in the database, this 1-hour period will be automatically grouped into one group. In a specific example, assuming the data from 8:00 to 9:00 has already been grouped into one group, then starting from 9:00, as new time-series data is continuously added to the database, when 10:00 arrives, the time-series data from 9:00 to 10:00 will be grouped into a new group. It should also be noted that in practice, the time is usually grouped by the hour, minute, and second. For example, one hour is grouped by the hour, such as 8:00, 9:00, and 10:00, rather than by the hour, such as 8:05, 9:05, and 10:05. Similarly, one minute is grouped by the minute, such as ...

[0102] In a specific implementation, the above method further includes: determining the matching index aggregation precision based on the cumulative storage time of the time-series data in the database; aggregating a corresponding number of intermediate indexes based on the index aggregation precision to obtain an aggregated intermediate index; wherein, the time slice corresponding to the aggregated intermediate index is the sum of the time slices corresponding to the corresponding number of intermediate indexes, and the operator calculation result corresponding to the aggregated intermediate index is the sum of the operator calculation results corresponding to the corresponding number of intermediate indexes.

[0103] Understandably, as time-series data accumulates in storage time, its "hotness" decreases. Therefore, this embodiment can aggregate multiple intermediate indexes to free up storage space, thereby achieving index lifecycle management. For example, the system can automatically roll up high-precision indexes at the "minute level" from the past 24 hours into low-precision indexes at the "hour level" in the background. For even earlier data, such as data from a month ago, it can be further aggregated into indexes at the "day level." That is, as data "ages," its index precision decreases, thus freeing up a significant amount of storage space. Furthermore, high-precision, hot data can be stored in high-speed storage, while low-precision, cold data can be archived to low-cost storage, thus forming a complete and automated index lifecycle management system.

[0104] Furthermore, when calculating intermediate state indices based on preset operators on time-series data within a group, these preset operators can include, but are not limited to, minimum, maximum, count, sum, average, and sum of squares. Moreover, multiple intermediate state indices can be established for time-series data within the same group using different operators, with each intermediate state index corresponding to one operator. For example, the time-series data from 8:00 to 9:00 can be calculated using both a count operator and a summation operator, resulting in two intermediate state indices. One intermediate state index has a time slice of 1 hour, and the corresponding operator calculates the total number of detailed data within that hour; the other intermediate state index has a time slice of 1 hour, and the corresponding operator calculates the sum of all detailed data within that hour.

[0105] Furthermore, for time-series data within the same group that uses at least two operators to create intermediate indexes, these intermediate indexes can be aggregated to obtain a time-series aggregated index (or aggregated summary). For example, if a time-series aggregated index is created with the following elements: (operators: {sum, count, maximum}, time shard: 1 hour), then the time-series aggregated index will contain the sum, total, and maximum value of all detailed data between 14:00 and 15:00.

[0106] Step S13: If a target intermediate state index that matches the search criteria exists in the database, then the corresponding search result is returned based on the target intermediate state index.

[0107] In this embodiment, if a target intermediate index is successfully matched in the database, the corresponding search result can be directly returned based on the target intermediate index. That is, since the intermediate index has been pre-built in this application, the operator calculation result can be directly reused when a match is successful, without having to repeatedly scan or calculate the original detailed data, significantly shortening the search response time and improving the efficiency of data retrieval and calculation. It can be seen that this application, for data with time-series characteristics in the database, pre-builds an intermediate index on the data through the time dimension, effectively solving the problem that the current calculation requires traversing detailed data, and improving the calculation speed.

[0108] As can be seen, this application pre-groups the time-series data in the database and calculates the time-series data within each group based on preset operators to establish an intermediate-state index. Each intermediate-state index includes the time slice of the corresponding group and the corresponding operator calculation result. When a data retrieval task is obtained, the retrieval conditions extracted from the data retrieval task are first matched with the established intermediate-state index. If a target intermediate-state index is successfully matched, the corresponding retrieval result can be directly returned based on the target intermediate-state index. That is, since this application has pre-established the intermediate-state index, the operator calculation result can be directly reused when a match is successful, without having to repeatedly scan or calculate the original detailed data, significantly shortening the retrieval response time and improving the efficiency of data retrieval and calculation. Therefore, this application, for this type of data with time-series characteristics in the database, pre-builds an intermediate-state index for the data through the time dimension, effectively solving the problem that the current calculation requires traversing detailed data and improving the calculation speed.

[0109] Furthermore, based on the aforementioned embodiments, the search conditions specifically include a search time range and a task operator type. For example, if the data search task is the total transaction amount of a user in the last 5 minutes, and assuming the current time is 9:00, then the search time range is extracted to be from 8:55 to 9:00, and the task operator type is the summation operator; as another example, if the data search task is the total number of transactions on the previous day, then the search time range is extracted to be from 00:00 to 24:00 on the previous day, and the task operator type is the counting operator.

[0110] In a first specific embodiment, matching the currently established intermediate indexes in the database with the search conditions includes: if any of the currently established intermediate indexes in the database has a time shard coverage range that is less than or equal to the search time range, and the calculation result of the operator corresponding to any of the intermediate indexes matches the task operator type, then the intermediate index is used as the target intermediate index that matches the search conditions; correspondingly, returning the corresponding search result based on the target intermediate index includes: constructing the corresponding search result based on the calculation result of the operator corresponding to the target intermediate index, and returning the search result.

[0111] It is understandable that when matching the currently established intermediate indexes in the database with the search conditions, the matching is mainly based on two aspects: the search time range and the task operator type.

[0112] Specifically, regarding the retrieval time range, the time range covered by the time shard corresponding to the intermediate index must be less than or equal to the retrieval time range. For example, if the current search is for data from 8:55 to 9:00, it is necessary to search for whether there is an intermediate index with a time shard of less than or equal to 5 minutes within the time period from 8:55 to 9:00. If the time shard of the pre-built intermediate index is 1 minute, then 5 intermediate indexes can be retrieved. If the time shard of the pre-built intermediate index is 5 minutes, then 1 intermediate index can be retrieved. If the time shard of the pre-built intermediate index is 10 minutes, then there is no intermediate index that meets the retrieval time range.

[0113] For the retrieval time range, the calculation result of the operator corresponding to the intermediate index needs to match the task operator type. For example, if the current task operator type is a summation operator, then it is necessary to check whether there is a summation result in the calculation result of the corresponding operator in the intermediate index.

[0114] Therefore, if among the currently established intermediate indexes in the database, there exists an intermediate index whose corresponding time shard covers a time range less than or equal to the retrieval time range, and the calculation result of the operator corresponding to that intermediate index also matches the task operator type, then that intermediate index is used as the target intermediate index matching the retrieval conditions. It can be understood that the number of target intermediate indexes matching the retrieval conditions can be one or more. It should also be noted that the matching process for the retrieval time range and the matching process for the task operator type can be performed simultaneously or sequentially; this embodiment does not impose any restrictions on this.

[0115] In this way, when returning the corresponding search results based on the target intermediate index, the search results can be constructed and returned based on the operator calculation results corresponding to the target intermediate index. In specific implementation, the operator calculation results of this intermediate index can be directly reused, which can quickly obtain the calculation results and improve the calculation speed, thus avoiding repeating the calculation process on the time-series data corresponding to the intermediate index. For example, if the data retrieval task is to query the total transaction amount from 8:00 to 10:00, then there are two intermediate indexes in the pre-built intermediate indexes: one for the total transaction amount from 8:00 to 9:00 and another for the total transaction amount from 9:00 to 10:00. The results of these two intermediate indexes can be directly reused and then added together to obtain the search results.

[0116] Furthermore, based on this, the search conditions also include a business dimension field for characterizing the business type of the data; the target group is a group obtained by dividing the time-series data in the database according to different business types; each intermediate index also includes the target business type of the target group; correspondingly, matching the currently established intermediate indexes in the database with the search conditions includes: determining whether there exists any intermediate index whose target business type matches the business dimension field in the search conditions; if so, filtering out the initial intermediate indexes that match the business dimension field in the search conditions, and matching the time slicing and operator calculation results corresponding to the initial intermediate indexes with the search time range and task operator type in the search conditions respectively, to determine whether there exists a target intermediate index that matches the search conditions in the initial intermediate indexes; if there is no intermediate index that matches the business dimension field in the search conditions, it is determined that there is no target intermediate index that matches the search conditions in the database.

[0117] Understandably, in some complex scenarios, each data point in a time-series dataset, in addition to its time attribute, also includes m business dimension fields. Specifically, a set of time-series datasets is... ,in, , Represents an m+1 dimensional real space. Each data point In addition to core time attributes (such as timestamps), it also includes m business dimension fields. The corresponding value. For example, in actual anti-telecom fraud scenarios, anti-fraud experts need to quickly execute the query "total number, total amount, and maximum amount of all 'interbank transfers' made from within China to 'a certain overseas region' during the time period from 14:05 to 14:35 today." Here, the time-series data set P is financial transaction log data, consisting of a series of data points with timestamps. The structure consists of each data point s, which, in addition to representing the timestamp of the transaction, has a time attribute. In addition, it also includes multiple business dimension attributes D={the recipient's location, the merchant IDs of both parties in the transaction, the transaction type, the transaction amount, etc.}. The data structure of the time-series data set P of the financial transaction flow data in this example is as follows: Figure 3 As shown.

[0118] Therefore, to enhance the analytical capabilities of the index, each target group can also be a group obtained by dividing the time-series data in the database according to different business types. That is, this application can build intermediate indexes on specific time shards and specified business types. Thus, in addition to the time attribute of time sharding, each target group can also include one or more target business types as business attributes, such as city and merchant ID in a transaction scenario, or host IP and application name in server monitoring. In this way, each intermediate index includes not only the time shard of the target group and the corresponding operator calculation results, but also the target business type of the target group.

[0119] In a specific implementation, let For fixed time segments (e.g., 1 hour, 10 minutes), the time is first divided into sets of time segments according to the time segments. Since the time slices in this set are fixed, they can be composed of a series of start timestamps for each time slice. Then, based on the time slice set T and the specified business type d, a combination of each time slice and dimension value is generated. Create an index entry to list the data points in the time series dataset P. according to Grouping to obtain a group set , where each group That is, a data shard corresponds to a unique , Can be defined as .

[0120] Furthermore, this application defines an intermediate index as a secondary key. to value Mapping. This involves constructing a key-value pair structure, where the key is a composite key. It consists of time sharding and target business type; the value is... This represents the result of operator calculations, also known as an "aggregate summary," which includes the grouping of the current key using the operator set O {minimum, maximum, count, sum, average, sum of squares}. The result of the pre-calculation, i.e. For example, in the anti-telecom fraud scenario mentioned above, an intermediate index was created to query the total transaction amount. Its elements are (operator: {sum}, time sharding: 10 minutes, business type: {recipient's location, transaction type}). Then, in this index, the entry with the key (14:00, {'overseas region', 'interbank transfer'}) will have an "aggregate summary" containing the total amount of transaction details for interbank transfers between 14:00 and 14:10 where the recipient's location is "overseas region".

[0121] Correspondingly, search criteria can also include business dimension fields that characterize the business type of the data, and each search can be defined as a tuple. When a search is initiated, the server receives the search task Q and extracts the task operator type required for the current task. Time range and business dimension fields When matching the currently established intermediate indexes in the database with the search criteria, it is advisable to first determine whether any of the currently established intermediate indexes contains a target business type that matches the business dimension field in the search criteria. If a match is found, and it does not exist, it is directly determined that there is no target intermediate index in the database that matches the search conditions. If it does exist, further filtering is performed based on the search time range and task operator type. The specific filtering process is the same as the aforementioned public content and will not be repeated here.

[0122] In some specific implementations, all candidate indexes that may serve this retrieval can be filtered from the system catalog based on the filtering conditions (i.e., time range and business dimension fields) in retrieval task Q and the aggregate function in SELECT. A cost model is then used to estimate the cost of using each candidate index, and finally, the candidate index with the lowest cost is selected as the matching index. For example, the final result is an index that best matches the business dimension and has the most suitable time granularity, minimizing computational cost while satisfying constraints. For the finally selected index, the executor calculates the range of composite keys that need to be scanned. Assuming the business dimension field in the retrieval conditions is a city name, such as "City A" or "City B", the executor will push this condition to the storage engine layer. This way, when scanning the index, index entries that do not belong to "City A" or "City B" can be skipped directly, avoiding unnecessary data reading and upper-level calculations.

[0123] Furthermore, after adding the business type, the elements of the time-series aggregated index are (operators: {sum, count, maximum value}, time sharding: 1 hour, target business type: city). Then, the index entry of the composite key (14:00-15:00, city A) will have a time-series aggregated index containing the sum, total, and maximum value of all detailed data for "city A" between 14:00 and 15:00.

[0124] In a second specific embodiment, when matching the currently established intermediate indexes in the database with the search conditions, the method further includes: if none of the currently established intermediate indexes in the database have an operator calculation result that matches the task operator type, and at least two intermediate indexes have operator calculation results that have a preset algorithm dependency relationship with the task operator type, then the at least two intermediate indexes are determined as target intermediate indexes that match the search conditions; wherein the at least two intermediate indexes correspond to the same time slice, have different operator calculation results, and the time range covered by the time slices corresponding to the at least two intermediate indexes is less than or equal to the search time range; accordingly, returning the corresponding search result based on the target intermediate index includes: calculating based on the operator calculation results corresponding to the at least two intermediate indexes according to the preset algorithm dependency relationship to obtain a target calculation result that matches the task operator type; constructing the corresponding search result based on the target calculation result, and returning the search result.

[0125] In other words, if none of the currently established intermediate indexes in the database contain an operator whose calculation result precisely matches the task operator type, it is also possible to query whether there exists an intermediate index in the current database that has a pre-defined algorithm dependency relationship with the task operator type, i.e., to query whether an operator already supports its corresponding mathematical algorithm. For example, suppose the current task operator type is averaging (AVG), and the operators with algorithmic dependencies on averaging are summation (SUM) and counting (COUNT), i.e., according to the formula... The average value can be obtained using summation and counting operators. If none of the currently established intermediate indexes in the database produce an average value, but two types of intermediate indexes exist—one type producing a count value and the other a summation value—and these two types of intermediate indexes share the same time slice (e.g., both are 1 hour), but their operator results differ (e.g., one a count and the other a summation), and the time slices of these two types of intermediate indexes cover a time range less than or equal to the retrieval time range, then these two types of intermediate indexes are considered target intermediate indexes matching the retrieval criteria.

[0126] In this way, when returning the corresponding search results based on the target intermediate index, the operator calculation results corresponding to these two intermediate indexes can be reused. Specifically, the operator calculation results corresponding to the two intermediate indexes are calculated according to a preset algorithm dependency relationship to obtain the target calculation result matching the task operator type. Then, the corresponding search results are constructed based on the target calculation result and returned. That is, when an existing index supports obtaining results through mathematical calculation and the time sharding also meets the requirements, the existing index can be reused. Compared with full data scanning, this method reduces the amount of computation and improves the calculation speed.

[0127] For example, if the data retrieval task is to query the average transaction amount during the period from 8:00 to 9:00, then in the pre-built intermediate index, there is one intermediate index for the total transaction amount during the period from 8:00 to 9:00 and another intermediate index for the total number of transactions during the period from 8:00 to 9:00. Then, the operator calculation results of these two intermediate indexes can be directly reused, and the calculation results of these two operator calculation results can be calculated according to the average method, that is, the total transaction amount / the total number of transactions, to obtain the average transaction amount during the period from 8:00 to 9:00.

[0128] Furthermore, in some special cases, such as when the current database only contains one intermediate index with a pre-defined algorithmic dependency on the task operator type (e.g., the current task operator type is averaging, but the database only contains the calculation results of the counting operator corresponding to averaging), although the final conclusion is that there is no target intermediate index that perfectly matches the search criteria, the counting result corresponding to the counting operator can still be reused when returning the final search result. That is, only one summation operation needs to be performed on all the detailed data within that time range, without performing another counting operation.

[0129] In a third specific embodiment, when matching the currently established intermediate indexes in the database with the search conditions, the method further includes: if none of the currently established intermediate indexes in the database have a time segment coverage range less than or equal to the search time range, then it is determined that there is no target intermediate index in the database that matches the search conditions; or, if any of the currently established intermediate indexes in the database have a time segment coverage range less than or equal to the search time range, and none of the intermediate indexes have an operator calculation result that matches the task operator type, and none of the at least two intermediate indexes have an operator calculation result that has a preset algorithm dependency relationship with the task operator type, then it is determined that there is no target intermediate index in the database that matches the search conditions.

[0130] In other words, during the process of matching the currently established intermediate indexes in the database with the search criteria, there are also situations where the intermediate indexes cannot be matched. This mainly includes the following two specific situations:

[0131] In a specific scenario, if all the time shards of the intermediate indexes currently established in the database cover a time range that is greater than the retrieval time range—for example, if the current retrieval is for the total transaction amount between 8:00 and 9:05, but all the time shards of the intermediate indexes are 10 minutes or more—then it can be directly determined that there is no target intermediate index in the database that matches the retrieval criteria.

[0132] In another specific case, if the intermediate indexes currently established in the database have time shards that cover a time range less than or equal to the required retrieval time range, but the calculation results of the operators corresponding to each intermediate index do not match the task operator type, and there is no preset algorithm dependency relationship between the calculation results of the operators corresponding to at least two intermediate indexes and the task operator type, then it is determined that there is no target intermediate index in the database that matches the retrieval conditions.

[0133] In other words, this embodiment specifically provides two situations in which intermediate state indexes cannot be matched. Through multi-level condition judgment, it is possible to accurately identify which situation the intermediate state index cannot be reused.

[0134] Furthermore, the above method also includes: if there is no target intermediate state index matching the search conditions in the database, then querying first time-series data that meets the search time range from the database; calculating the first time-series data using the task operator type to obtain a first calculation result, and returning the search result constructed based on the first calculation result.

[0135] In other words, when no target intermediate index matching the search criteria exists in the database, the existing intermediate index cannot be reused. Instead, the detailed data must be calculated using traditional methods. This involves querying the database for first-time-series data matching the search time range, then using task operator types to calculate the first-time-series data to obtain the corresponding first calculation result. Finally, the search result constructed based on the first calculation result is returned. Specifically, as follows... Figure 4 As shown. Furthermore, at this point, users can choose whether to create a new intermediate index based on this operation, thereby enabling dynamic updates to the intermediate index.

[0136] In addition, it should be noted that when a user creates a new intermediate index, if the same operator already exists in an existing intermediate index and its time shard is less than or equal to the time shard of the new index, the server can suggest reusing the existing index without creating a new index.

[0137] Furthermore, based on the aforementioned embodiments, each intermediate index also includes the start time of the time-series data within the target group; correspondingly, determining the time range covered by the time slice corresponding to any intermediate index includes: determining the time range covered by any intermediate index based on the start time and time slice corresponding to any intermediate index.

[0138] Understandably, intermediate indexes, in addition to including time shards and corresponding operator calculation results, can also include the start time of time-series data within the target group. These intermediate indexes can be constructed using a key-value pair structure, where, without considering business dimension fields, the key corresponds to the start time and time shard of the shard, and the value corresponds to the operator calculation result.

[0139] For details, please refer to [link / reference]. Figure 5 As shown, for the period from 15:00 to 16:00, two intermediate indexes are established, both starting at 15:00. One intermediate index has a time slice of 1 hour, and its operator calculation result is a summation result. The other intermediate index also has a time slice of 1 hour, and its operator calculation result is a count result.

[0140] Specifically, in this embodiment, when comparing the time range covered by the time slice corresponding to the intermediate index with the retrieval time range, the time range covered by the intermediate index can be calculated first based on the start time and time slice corresponding to the intermediate index, and then the two can be compared. For example, using... Figure 5For example, if the starting time of the intermediate index is 15:00 and the time shard is 1 hour, then its covered time range is 15:00-16:00. If the retrieval time range of the current data retrieval task is 15:00-17:00, then since 15:00-16:00 is within 15:00-17:00, the intermediate index can be reused. However, if the retrieval time range of the current data retrieval task is 15:00-15:30, then the intermediate index cannot be reused.

[0141] Based on the foregoing embodiments, the method of this application further includes: if the sum of the time ranges covered by each of the target intermediate state indexes is less than the retrieval time range, then determining the remaining time range not covered by the target intermediate state index; querying second time-series data that conforms to the remaining time range from a preset index structure tree, and calculating the second time-series data using the task operator type to obtain a second calculation result; wherein, the index structure tree is constructed based on all time-series data and intermediate state indexes in the database; correspondingly, returning the corresponding retrieval result based on the target intermediate state index includes: summarizing the target intermediate state index and the second calculation result to construct the corresponding retrieval result, and returning the retrieval result.

[0142] Understandably, in practice, the time ranges covered by each intermediate index are not perfectly aligned with the retrieval time range. For example, suppose the current query is for data between 8:30 and 10:00, and the time shard size of the intermediate index built for the data between 8:00 and 17:00 is 1 hour. This means that in this case, only the intermediate index corresponding to 9:00-10:00 can be matched. Therefore, when the sum of the time ranges covered by each target intermediate index is less than the retrieval time range, it is necessary to determine the remaining time range not covered by the target intermediate indexes. In the aforementioned example, the remaining time range is 8:30-9:00. For the data required for the remaining time range, this embodiment also needs to calculate the detailed data according to the traditional solution, that is, query the time-series data that matches the remaining time range from the database, and then use the task operator type to calculate the corresponding calculation results. In this way, the index can be dynamically reused, and the data within the remaining time range can be supplemented as needed, thereby achieving accurate support for queries within any retrieval time range.

[0143] It is important to note that, in order to improve the query efficiency of data within the remaining time range, this embodiment constructs an index structure tree based on all time-series data and intermediate indexes in the database. The entire index structure tree is stored as a B-tree, and each leaf node in the index structure tree corresponds to a time-series data record. Leaf nodes with adjacent time sequences are connected by pointers, as detailed below. Figure 5As shown in the diagram. Therefore, when querying detailed data, the second time-series data that matches the remaining time range can be retrieved from the index tree structure. The second time-series data is then calculated using the task operator type to obtain the second calculation result. Since all leaf nodes of the B-tree structure are ordered linked lists, their sorting characteristic is time order. When the upper and lower boundaries of the target intermediate index are located, only one node forward and / or to the right of the upper and lower boundaries needs to be retrieved to obtain the complete calculation data within the retrieval time range, thus reducing the time complexity of the query. Furthermore, when the intermediate index also includes the target business type, it is sorted in the B-tree structure according to the business type and timestamp order to achieve efficient scanning of the time range under a specific dimension. In this way, the second calculation result corresponding to the remaining time range is then aggregated with the target intermediate index as a temporary intermediate index according to the time order to obtain the complete retrieval result.

[0144] Specifically, firstly, based on the time range Find the upper and lower boundaries of the corresponding current time series intermediate state index. ,in , Defined as retrieving data from the upper and lower boundaries one node forward and one node backward that meets the time range. Detailed data, Defined as in There is no need to read intermediate state indexes containing detailed data. For example, there are currently five intermediate state indexes S1, S2, S3, S4, and S5, and each index contains two detailed data entries. and Where x represents the index index, and assuming the retrieval time range is... This time interval can be used to confirm the intermediate indexes contained within the upper and lower boundaries of the current intermediate index. The detailed data that needs to be read is When performing data retrieval calculations, detailed data can be found by searching within a specific time range. With intermediate index By recalculating the detailed data into a temporary intermediate index and then aggregating it with the target intermediate index, the final retrieval result r can be quickly obtained. The final result r can be defined as... Here, Agg represents the aggregation method, such as summing the SUM results of each partition or taking the maximum of the MAX results of each partition. As in the example above, , ,but The flowchart for the result calculation is as follows: Figure 6 As shown.

[0145] As can be seen, this application introduces the concept of an intermediate-state index, which can be built on a millisecond-level timestamp field. Through this intermediate-state index, the index can be dynamically aggregated based on the index status during retrieval / calculation, accelerating the calculation speed and effectively solving the problem of needing to traverse detailed data during calculation, reducing the amount of computation and improving the calculation speed. Furthermore, through the technical solution of this invention, intermediate-state indexes with different time shards can be reused in different retrieval time ranges, greatly reducing the creation of indexes, alleviating the pressure on server storage and retrieval, and improving the index reuse rate.

[0146] See Figure 7 As shown in the figure, this application discloses a data retrieval device based on a time-series intermediate state, the device comprising:

[0147] Condition extraction module 11 is used to extract search conditions from data retrieval tasks;

[0148] The judgment module 12 is used to match the currently established intermediate index in the database with the search conditions; the intermediate index is an index established based on the calculation of time series data in the target group by a preset operator, the target group is a group obtained by dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result;

[0149] The result return module 13 is used to return the corresponding search result based on the target intermediate index if a target intermediate index that matches the search conditions exists in the database.

[0150] As can be seen, this application pre-groups the time-series data in the database and calculates the time-series data within each group based on preset operators to establish an intermediate-state index. Each intermediate-state index includes the time slice of the corresponding group and the corresponding operator calculation result. When a data retrieval task is obtained, the retrieval conditions extracted from the data retrieval task are first matched with the established intermediate-state index. If a target intermediate-state index is successfully matched, the corresponding retrieval result can be directly returned based on the target intermediate-state index. That is, since this application has pre-established the intermediate-state index, the operator calculation result can be directly reused when a match is successful, without having to repeatedly scan or calculate the original detailed data, significantly shortening the retrieval response time and improving the efficiency of data retrieval and calculation. Therefore, this application, for this type of data with time-series characteristics in the database, pre-builds an intermediate-state index for the data through the time dimension, effectively solving the problem that the current calculation requires traversing detailed data and improving the calculation speed.

[0151] Since the embodiments of the device part correspond to the embodiments described above, please refer to the embodiments described in the method part for the embodiments of the device part, and will not be repeated here.

[0152] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Specifically, it may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the time-series intermediate state-based data retrieval method disclosed in any of the foregoing embodiments.

[0153] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0154] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0155] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.

[0156] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the time-series intermediate state-based data retrieval method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the electronic device from external devices, as well as data collected by its own input / output interface 25.

[0157] Furthermore, embodiments of this application also disclose a computer-readable storage medium storing a computer program. When the computer program is loaded and executed by a processor, it implements the data retrieval method steps based on time-series intermediate states disclosed in any of the foregoing embodiments.

[0158] This invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the data retrieval method based on temporal intermediate states disclosed in any of the foregoing embodiments.

[0159] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0160] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, compact disc read-only memory (CD-ROM), or any other form of storage medium known in the art.

[0162] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0163] The present invention provides a detailed description of a data retrieval method, apparatus, device, medium, and product based on a time-series intermediate state. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data retrieval method based on temporal intermediate states, characterized in that, include: Extract search criteria from the data retrieval task; wherein, the search criteria include the search time range and the task operator type; The intermediate index currently established in the database is matched with the search conditions; the intermediate index is an index established based on the calculation of time series data in the target group by a preset operator, the target group is a group obtained by dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result; If a target intermediate state index matching the search criteria exists in the database, the corresponding search result is returned based on the target intermediate state index. The method further includes: If none of the intermediate indexes currently established in the database have an operator calculation result that matches the task operator type, and at least two intermediate indexes have operator calculation results that have a preset algorithm dependency relationship with the task operator type, then the at least two intermediate indexes are determined as target intermediate indexes that match the retrieval conditions; the at least two intermediate indexes have the same time slice, different operator calculation results, and the time range covered by the time slices of the at least two intermediate indexes is less than or equal to the retrieval time range.

2. The data retrieval method based on time-series intermediate states according to claim 1, characterized in that, The step of matching the currently established intermediate index in the database with the search conditions includes: If any intermediate index in the database currently exists has a time shard that covers a time range less than or equal to the retrieval time range, and the calculation result of the operator corresponding to any intermediate index matches the task operator type, then that intermediate index will be used as the target intermediate index that matches the retrieval conditions. Accordingly, returning the corresponding search results based on the target intermediate index includes: The corresponding search results are constructed based on the operator calculation results corresponding to the target intermediate state index, and the search results are returned.

3. The data retrieval method based on time-series intermediate states according to claim 1, characterized in that, The return of corresponding search results based on the target intermediate index includes: The calculation results of the operators corresponding to the at least two intermediate state indices are calculated according to the preset algorithm dependency relationship to obtain the target calculation result that matches the task operator type; Based on the target calculation results, construct the corresponding search results and return the search results.

4. The data retrieval method based on time-series intermediate states according to claim 1, characterized in that, Also includes: If none of the intermediate indexes currently established in the database cover a time range less than or equal to the retrieval time range, then it is determined that there is no target intermediate index in the database that matches the retrieval conditions. Alternatively, if any intermediate index in the database currently has a time shard covering a time range less than or equal to the retrieval time range, and no intermediate index has an operator calculation result that matches the task operator type, and no two intermediate indexes have operator calculation results that have a preset algorithm dependency relationship with the task operator type, then it is determined that there is no target intermediate index in the database that matches the retrieval conditions.

5. The data retrieval method based on time-series intermediate states according to claim 1, characterized in that, Also includes: If no target intermediate state index matching the search criteria exists in the database, then first time-series data that meets the search time range is queried from the database. The task operator type is used to calculate the first time series data to obtain a first calculation result, and the retrieval result constructed based on the first calculation result is returned.

6. The data retrieval method based on time-series intermediate states according to claim 1, characterized in that, The search criteria also include a business dimension field for characterizing the business type of the data; the target group is a group obtained by dividing the time-series data in the database according to different business types; each intermediate index also includes the target business type of the target group; Accordingly, matching the currently established intermediate indexes in the database with the search conditions includes: Determine whether any of the currently established intermediate indexes in the database matches the target business type of the business dimension field in the search criteria. If it exists, the initial intermediate index that matches the business dimension field in the search conditions is selected, and the time slicing and operator calculation results corresponding to the initial intermediate index are matched with the search time range and task operator type in the search conditions respectively to determine whether there is a target intermediate index that matches the search conditions in the initial intermediate index. If no intermediate index matches the business dimension field in the search criteria, it is determined that no target intermediate index matches the search criteria in the database.

7. The data retrieval method based on time-series intermediate states according to claim 2, characterized in that, Each intermediate index also includes the start time of the time-series data within the target group; Accordingly, the time range covered by the time shard corresponding to any intermediate index is determined, including: The time range covered by any intermediate index is determined based on the start time and time slice corresponding to any intermediate index.

8. The data retrieval method based on time-series intermediate states according to claim 7, characterized in that, Also includes: If the sum of the time ranges covered by each of the target intermediate indexes is less than the retrieval time range, then the remaining time ranges not covered by the target intermediate indexes are determined. The second time-series data that matches the remaining time range is queried from the preset index structure tree, and the second time-series data is calculated using the task operator type to obtain a second calculation result; wherein, the index structure tree is constructed based on all time-series data and intermediate state indexes in the database; Accordingly, returning the corresponding search results based on the target intermediate index includes: The target intermediate state index and the second calculation result are summarized to construct the corresponding search result, and the search result is returned.

9. The data retrieval method based on time-series intermediate states according to claim 8, characterized in that, Each leaf node in the index structure tree corresponds to a time-series data, and leaf nodes with adjacent time sequences are connected by pointers.

10. The data retrieval method based on time-series intermediate states according to any one of claims 1 to 9, characterized in that, The process of dividing and grouping the time-series data in the database includes: The time-series data in the database is divided into several groups according to fixed time segments; Alternatively, the historical time-series data stored in the database can be analyzed to determine the data density corresponding to different time periods, so as to establish a correspondence between time periods and time slice sizes based on the data density; wherein the data density and the time slice size are negatively correlated. Determine the target time period to which the time series data to be divided belongs, and divide the time series data to be divided into several groups using the target time segment size corresponding to the target time period.

11. The data retrieval method based on time-series intermediate states according to any one of claims 1 to 9, characterized in that, Also includes: The matching index aggregation precision is determined based on the cumulative storage time of the time-series data in the database; Based on the index aggregation precision, a corresponding number of intermediate state indices are aggregated to obtain aggregated intermediate state indices; wherein, the time slice corresponding to the aggregated intermediate state index is the sum of the time slices corresponding to the corresponding number of intermediate state indices, and the operator calculation result corresponding to the aggregated intermediate state index is the sum of the operator calculation results corresponding to the corresponding number of intermediate state indices.

12. A data retrieval device based on a time-series intermediate state, characterized in that, include: The condition extraction module is used to extract search conditions from the data retrieval task; wherein, the search conditions include the search time range and the task operator type; The judgment module is used to match the currently established intermediate indexes in the database with the search conditions; the intermediate index is an index established based on the calculation of time series data in the target group by a preset operator, the target group is a group obtained by dividing the time series data in the database, and each intermediate index includes the time slice of the target group and the corresponding operator calculation result; The result return module is used to return the corresponding search result based on the target intermediate index if a target intermediate index that matches the search conditions exists in the database. The device is further configured to, if in the currently established intermediate indexes of the database, there is no intermediate index whose operator calculation result matches the task operator type, and there are at least two intermediate indexes whose operator calculation results have a preset algorithm dependency relationship with the task operator type, then determine the at least two intermediate indexes as target intermediate indexes that match the retrieval conditions; the at least two intermediate indexes correspond to the same time slice, but have different operator calculation results, and the time range covered by the time slices corresponding to the at least two intermediate indexes is less than or equal to the retrieval time range.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data retrieval method based on a time-series intermediate state as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the steps of the data retrieval method based on any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the data retrieval method based on any one of claims 1 to 11.

Images