Data query method, device, apparatus, and storage medium

Abstract

Embodiments of the present disclosure provide a data query method, a data query method for a distributed system, and corresponding apparatuses, devices and storage media. The method provided by the embodiments of the present disclosure removes indexes irrelevant to a data query request from an index group by performing index filtering according to specific data interval range information in the data query request and data interval range information of indexes in the index group, and then searches for data records corresponding to the specific data interval range information in the data query request from the filtered index group by performing a shard query, thereby achieving efficient data query. The method of the embodiments of the present disclosure can avoid sending a query request to shards of a large number of indexes having no data intersection in a query phase by query interval filtering, thereby greatly reducing query latency and invalid query load of a cluster, and improving user experience.

Description

Technical Field

[0001] This disclosure relates to the field of cloud technology, and more specifically, to a data query method, a data query method for a distributed system, and corresponding apparatus, devices and storage media. Background Technology

[0002] Elasticsearch, a distributed search and analytics engine, is a search server based on Lucene (an open-source full-text search engine toolkit). It provides a distributed, multi-user full-text search engine. Commonly used in cloud computing, Elasticsearch enables real-time searching, is fast, stable, reliable, and easy to install and use. It can be used to search various documents, enabling fast and efficient querying and aggregation of large amounts of data. In Elasticsearch, data is stored in shards; during a search, each shard is searched independently, and the retrieved data is then integrated and returned.

[0003] Elasticsearch, a distributed, highly scalable, and real-time search and data analytics engine, provides near real-time search and analysis for all types of data. Whether it's structured or unstructured text, numerical data, or geospatial data, Elasticsearch efficiently stores and indexes it in a way that supports fast search. While Elasticsearch's distributed nature allows for seamless deployment growth as data and query volume increase, it also presents challenges to data query performance.

[0004] Therefore, an optimized data query method is needed to enable efficient data querying in Elasticsearch. Summary of the Invention

[0005] To address the aforementioned issues, this disclosure utilizes specific data range information within the data query request to filter query ranges and remove unnecessary query ranges, thereby achieving query pruning and optimizing query performance.

[0006] The embodiments of this disclosure provide a data query method, a data query method for a distributed system, and corresponding apparatus, devices, and storage media.

[0007] This disclosure provides a data query method, comprising: receiving a data query request for an index group, the data query request including specific data range information; obtaining index metadata of the indexes in the index group; performing index filtering on the index group based on the obtained index metadata to determine a subset index group of the index group, wherein the index metadata of each index in the subset index group includes data range information that overlaps with the specific data range information; and performing a shard query on each index in the subset index group to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request.

[0008] This disclosure provides a data query method for a distributed system, the distributed system including a coordinating node and data nodes. The method includes: the coordinating node receiving a data query request for an index group, the data query request including specific data range information; the coordinating node obtaining index metadata of the indexes in the index group; the coordinating node performing index filtering on the index group based on the obtained index metadata to determine a subset index group, the index metadata of each index in the subset index group including data range information that overlaps with the specific data range information; and the coordinating node sending a shard query request to the data node where the shard of each index in the subset index group is located, and the data node performing a shard query based on the shard query request to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request.

[0009] Embodiments of this disclosure provide a data query apparatus, comprising: a request receiving module configured to receive a data query request for an index group, the data query request including specific data range information; a data acquisition module configured to acquire index metadata of the indexes in the index group; an index filtering module configured to perform index filtering on the index group based on the acquired index metadata to determine a subset index group of the index group, wherein the index metadata of each index in the subset index group includes data range information that overlaps with the specific data range information; and a shard query module configured to perform shard query on each index in the subset index group to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group as the query response result of the data query request.

[0010] Embodiments of this disclosure provide a data query device, including: one or more processors; and one or more memories, wherein the one or more memories store a computer-executable program, and when the processor executes the computer-executable program, it performs the data query method as described above.

[0011] Embodiments of this disclosure provide a computer-readable storage medium having computer-executable instructions stored thereon, which, when executed by a processor, are used to implement the data query method described above.

[0012] Embodiments of this disclosure provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data query method according to embodiments of this disclosure.

[0013] Compared to existing data query methods, the method provided by the embodiments of this disclosure can find partial indexes and partial fragments associated with user data requests from a large number of indexes, thereby improving the efficiency of data query and reducing the time consumption of data query.

[0014] The method provided in the embodiments of this disclosure performs index filtering based on specific data range information in the data query request and data range information of the indexes in the index group. This removes indexes in the index group that are irrelevant to the data query request. Then, it searches for data records corresponding to the specific data range information in the data query request from the filtered index group through sharded queries, thereby achieving efficient data querying. The method in the embodiments of this disclosure avoids sending query requests to shards of a large number of indexes with no data overlap during the query phase by filtering query ranges, thus greatly reducing query latency and cluster invalid query load, and improving user experience. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some exemplary embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0016] Figure 1A This is a schematic diagram illustrating a scenario in which an input data query request is processed and a query response result is returned according to an embodiment of the present disclosure;

[0017] Figure 1B This is a schematic diagram illustrating the query and crawling phases in a conventional query method according to an embodiment of the present disclosure.

[0018] Figure 2A This is a flowchart illustrating a data query method according to an embodiment of the present disclosure;

[0019] Figure 2B This is a flowchart illustrating a data query method for a distributed system according to an embodiment of the present disclosure;

[0020] Figure 3 This is a schematic diagram illustrating index groups, indexes, and fragments according to embodiments of the present disclosure;

[0021] Figure 4 This is a schematic flowchart illustrating the process of obtaining index metadata of an index in an index group according to an embodiment of the present disclosure;

[0022] Figure 5 This is a timing diagram illustrating the entire data query process according to an embodiment of the present disclosure;

[0023] Figure 6 This is a schematic flowchart illustrating the index filtering stage in a data query process according to an embodiment of the present disclosure;

[0024] Figure 7 This is a schematic diagram illustrating a data query apparatus according to an embodiment of the present disclosure;

[0025] Figure 8 A schematic diagram of a data query device according to an embodiment of the present disclosure is shown;

[0026] Figure 9 A schematic diagram of the architecture of an exemplary computing device according to embodiments of the present disclosure is shown; and

[0027] Figure 10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0029] In this specification and accompanying drawings, steps and elements that are substantially the same or similar are indicated by the same or similar reference numerals, and repeated descriptions of these steps and elements are omitted. Furthermore, in the description of this disclosure, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0031] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0032] The data query method disclosed herein can be based on cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of technical network systems require a large amount of computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the Internet industry, in the future, every item may have its own identification mark, which will need to be transmitted to the backend system for logical processing. Data of different levels will be processed separately, and various industry data will all require strong system support, which can only be achieved through cloud computing. In the data query method disclosed herein, data can be queried on multiple machines within a WAN or LAN using query range filtering operations based on cloud technology to obtain data records corresponding to the data query request.

[0033] The data query method disclosed herein can be based on the distributed search and analysis engine Elasticsearch. Elasticsearch is a distributed, highly scalable, and real-time search and data analysis engine. It can easily enable large amounts of data to be searched, analyzed, and explored. The implementation principle of Elasticsearch mainly consists of the following steps: First, the user submits data to the Elasticsearch database. Then, the tokenization controller segments the corresponding sentences into tokens, stores the weights and tokenization results together in the database. When the user searches for data, the results are ranked and scored according to the weights, and then the results are presented to the user. Elasticsearch can be used to search various documents. It provides scalable and near real-time search. Elasticsearch is distributed, which means that the index can be divided into shards, and each shard can have zero or more replicas. Each node hosts one or more shards and acts as a coordinator to delegate operations to the correct shard. In Elasticsearch, rebalancing and routing are done automatically, and related data are usually stored in the same index, which consists of one or more primary shards and zero or more replica shards. Once the index is created, the number of primary shards cannot be changed.

[0034] The following is a brief introduction to some concepts in the distributed search and analysis engine Elasticsearch.

[0035] Document / Data Record: A data record, such as a user access log.

[0036] A data segment is a set of immutable indexes and data files generated for data records produced within a specific continuous time window. Each data segment is an independent data storage unit that can support any query from the user.

[0037] Index: A container for storing documents, similar to a table in a traditional database.

[0038] Index Shard: The horizontal partitioning unit of an index. Shards are typically distributed across different machines to achieve horizontal data scaling.

[0039] Index Group: Multiple indexes created according to a predefined time period. Multiple indexes in the same index group have the same data structure definition.

[0040] Index metadata: The metadata describes information such as index settings, sharding routes, and data ranges for certain fields.

[0041] Shard metadata: The metadata describes the sharding settings, storage directory, and data range of some fields for this shard.

[0042] Data record segment metadata: The metadata describes the constituent files of this segment, as well as information such as the data range of some fields.

[0043] Index Rollover: Creates a new index and moves writes to the newly created index.

[0044] Master Node: The master node of the Elasticsearch cluster, used to create and distribute cluster metadata information.

[0045] Data Node: Used to store index shards.

[0046] Ingest Node: Used to receive user query write requests and is responsible for distributing and aggregating them.

[0047] In summary, the solutions provided by the embodiments of this disclosure involve technologies such as cloud technology and distributed system query. The embodiments of this disclosure will be further described below with reference to the accompanying drawings.

[0048] Figure 1A This is a schematic diagram illustrating a scenario in which an input data query request is processed and a query response result is returned according to an embodiment of the present disclosure. Figure 1B This is a schematic diagram illustrating the query and crawling phases in a conventional query method according to an embodiment of the present disclosure.

[0049] like Figure 1A As shown, a user terminal can send a data query request to the server via the network, so that the server can search for query response results corresponding to the information in the data query request. Subsequently, the server can return the determined query response results to the user terminal via the network for display to the user.

[0050] Optionally, the user terminal may specifically include smartphones, tablets, laptops, desktop computers, in-vehicle terminals, wearable devices, etc., but is not limited to these. The user terminal may also be a client with a browser or various applications (including system applications and third-party applications) installed. Optionally, the network may be an Internet of Things (IoT) based on the Internet and / or telecommunications networks. It can be a wired network or a wireless network, for example, it may be a local area network (LAN), metropolitan area network (MAN), wide area network (WAN), cellular data communication network, or other electronic networks capable of information exchange. The user terminal and the server can be directly or indirectly connected through wired or wireless communication, and this application does not impose any restrictions. The server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0051] exist Figure 1A The server shown can retrieve documents related to a user's query request from the system. For example, in a distributed system, the received query request can be forwarded to various nodes in the distributed system for independent querying, and the query results from each node can be integrated to obtain the final query response returned to the user's terminal.

[0052] For example, the data query method disclosed herein can be taken as an example of the distributed search and analysis engine Elasticsearch, and optimized for the data query method in that distributed system to achieve better data query performance. Therefore, although most of the descriptions in the data query method disclosed herein are based on that distributed system, this does not limit the application scenarios of the data query method disclosed herein. The data query method disclosed herein can also be applied to other distributed systems to optimize their data query performance.

[0053] For the distributed system Elasticsearch, existing query methods include: Figure 1B The diagram shows two phases: the query phase (...) Figure 1B (a) and the fetch stage ( Figure 1B (b) in the middle.

[0054] During the query phase, such as Figure 1B As shown in (a), the user terminal can send a data query request (such as...) to any coordinating node in the distributed system (e.g., node 1). Figure 1B (as shown by reference numeral ① in figure (a)), and the coordinating node that receives the data query request can broadcast the data query request to all data nodes where the index shards reside (e.g., nodes 2 and 3) (e.g. Figure 1B (As shown by reference numeral ② in figure (a)). Next, the data node that receives the data query request can execute a sharded query locally to search for the document ID and specific sort value that match the data query request and return the value to the coordinating node (e.g., node 1) (as shown in figure (a)). Figure 1B (as shown by reference numeral ③ in figure (a)).

[0055] Next, in the crawling phase, such as Figure 1B As shown in (b), the aforementioned coordinating node can sort the list of document IDs obtained during the query phase based on a specific sorting value, and, in conjunction with the field names specified in the data query request (e.g., timestamp field), send a document fetching request (e.g., ...) to the node where the document belongs (e.g., node 2 and node 3). Figure 1B (as shown by reference numeral ① in the attached diagram in (b)) to retrieve the document content. Upon receiving a document retrieval request, the node containing the shard can retrieve the corresponding document from the shard and return the document retrieval result to the aforementioned coordinating node (such as...). Figure 1B (As shown by reference numeral ② in (b) of the figure). Finally, the aforementioned coordinating node can integrate the document crawling results obtained from multiple data nodes into a query response result and return it to the user terminal (e.g., Figure 1B (as indicated by reference numeral ③ in (b) of the figure).

[0056] As mentioned above, existing Elasticsearch query methods in distributed systems require first querying all shards for document IDs that meet the query conditions, then aggregating and sorting these document IDs on a coordinating node, and finally retrieving document data from each data node based on the selected document IDs. This query method is a general query scheme for distributed Elasticsearch systems and works well when the number of shards in the target index is small. However, in scenarios such as log recording, indexes are typically rolled over daily, resulting in a large number of indexes scanned by queries. Furthermore, users usually only need to query recent data, causing the query phase to send query requests to a large number of indexes with no data overlap. This increases query latency and the cluster's ineffective query load, reduces query performance, and significantly impacts user experience.

[0057] Based on this, this disclosure provides a data query method that, based on specific data range information in a data query request, filters the query range to remove unnecessary query ranges in the data query, thereby achieving query pruning and optimizing query performance.

[0058] Compared to existing data query methods, the method provided by the embodiments of this disclosure can find partial indexes and partial fragments associated with user data requests from a large number of indexes, thereby improving the efficiency of data query and reducing the time spent on data query.

[0059] The method provided in the embodiments of this disclosure performs index filtering based on specific data range information in the data query request and data range information of the indexes in the index group. This removes indexes in the index group that are irrelevant to the data query request. Then, it searches for data records corresponding to the specific data range information in the data query request from the filtered index group through sharded queries, thereby achieving efficient data querying. The method in the embodiments of this disclosure avoids sending query requests to shards of a large number of indexes with no data overlap during the query phase by filtering query ranges, thus greatly reducing query latency and cluster invalid query load, and improving user experience.

[0060] Figure 2A This is a flowchart illustrating a data query method 200 according to an embodiment of the present disclosure. Figure 2B This is a flowchart illustrating a data query method 300 for a distributed system according to an embodiment of the present disclosure. As described above, the data query method 300 for a distributed system can be viewed as an application of the data query method 200 in a distributed system (e.g., Elasticsearch), wherein the distributed system may include the aforementioned master node, coordinating node, and data nodes. Therefore, the specific operation of the data query method of the present disclosure will be described below in conjunction with the data query method 200 and the data query method 300 for a distributed system.

[0061] like Figure 2A As shown, in step 201, a data query request for an index group can be received, and the data query request may include specific data range information.

[0062] The following is a combination of... Figure 3 The structure of the index group disclosed herein is described, wherein Figure 3 This is a schematic diagram illustrating index groups, indexes, and fragments according to embodiments of the present disclosure.

[0063] According to embodiments of this disclosure, each index in the index group may include one or more shards, wherein each shard may include one or more data record segments, and each data record segment may include one or more data records.

[0064] Optionally, the index group of this disclosure may include multiple indexes, each of which may store related data. For example, in scenarios such as log recording, indexes are typically rolled over daily, so each index may correspond to a data set in units of one day (or other time metric), and the index group may include multiple indexes corresponding to multiple consecutive time periods. Figure 3 As shown, the index group may include i indexes (index 1, ..., index i-1 and index i), where, in the above scenario, index 1 to index i may correspond to system data (e.g., data records, such as user access log information) for i consecutive days.

[0065] Optionally, the data in each index can be distributed across different nodes in the distributed system for storage; that is, the data in each index can be distributed across its multiple shards (e.g., Figure 3 The indexes shown all have three shards, which can reside on different machines (i.e., different data nodes in a distributed network) to achieve horizontal scaling of data in a distributed system. By storing data in shards within the index, parallel operations can be performed during data writing or querying, i.e., reading and writing data on various data nodes, improving throughput. Furthermore, within each shard, data can be stored as data record segments. Each data record segment is an independent data storage unit that supports any user query; it can be a set of immutable indexes and data files generated for data records (documents) produced within a specific continuous time window.

[0066] According to embodiments of this disclosure, the index group may include a latest index and read-only indexes, wherein the latest index is the latest index currently receiving new data writes, and the read-only indexes are the other indexes in the index group besides the latest index. In embodiments of this disclosure, the read-only indexes no longer receive new data writes, and the index metadata of the read-only indexes can be cached for data queries. The cached index metadata of the read-only indexes may include data range information of the read-only indexes, and the data range information of the latest indexes is not cached for data queries until the next write index switching event occurs in the index group. The latest indexes may be changed to read-only indexes in response to the next write index switching event.

[0067] Optionally, because index rolling may occur within the index group—that is, a new index is created and data writes are transferred to the newly created index—this newly created index can be responsible for receiving new data writes, while all previous indexes will no longer receive new data writes and will be marked as "read-only." In the index group, all indexes except the latest one currently receiving new data writes are read-only indexes, for example, in... Figure 3 In the index group shown, the current latest index can be index i, and all other indexes from 1 to i-1 are read-only indexes. When an index rollover occurs again in this index group, a new index i+1 (not shown) can be created in the group and receive new data writes. Therefore, index i+1 becomes the latest index in the index group, while the previous latest index i no longer receives new data writes and becomes a read-only index. Thus, since the data of a read-only index is fixed and cannot be modified, its index metadata can be cached in each coordinating node of the distributed system for use in subsequent data queries.

[0068] In embodiments of this disclosure, the metadata of each index, shard, and data record segment can include data range information with specific field names. According to embodiments of this disclosure, the data range information can be indicated by specific field name information, which can be associated with the data query request. Optionally, the specific field name information can be a predefined field set for query range filtering in this disclosure (the predefined field can be specified by the user through configuration when creating an index group), and the data range information can correspond to statistical information of the specific field name information. For example, for a data query request based on a time range, the specific field name information can be the timestamp information of the data (such as the timestamp of each data record), and the data range information can be a statistical analysis of the timestamp information of the data.

[0069] For example, the data range information of a data record segment can be determined by statistically analyzing the timestamp information of all the data records it includes, thereby defining the timestamp range of these data records as the data range information of the data record segment. According to embodiments of this disclosure, the data range information of each data record segment can be synchronously calculated based on the specific field name information of the data record segment during the generation of the data record segment. Optionally, the data range information of the specific field name information can be stored in the metadata information of the data record segment after the data writing of the data record segment is completed.

[0070] According to embodiments of this disclosure, the data range information of each shard can be determined based on the data range information of all data record segments in the shard, and the data range information of each index can be determined based on the data range information of all shards in the index. As described above, similar to the data range information of data record segments, the data range information of each shard can be a statistical analysis of the data range information of all data record segments it includes, thereby determining the timestamp range covered by these data record segments as the data range information of that shard. Similarly, the data range information of each index can be a statistical analysis of the data range information of all shards it includes, thereby determining the timestamp range covered by these shards as the data range information of that shard. Therefore, the index metadata of a cached read-only index can include the data range information of the read-only index, while the latest index, because it is still receiving new data writes, cannot determine its data range information before becoming a read-only index. Therefore, the data range information of the latest index is not cached for data querying before the next write index switch event occurs in the index group.

[0071] As described above, in the embodiments of this disclosure, query range filtering can be performed based on the specific data range information included in the user's data query request and the various data range information in the index group, thereby improving query performance. Specifically, as shown in step 301, a coordinating node can receive a data query request for the index group, and the data query request may include specific data range information. This specific data range information in the data query request can be extracted by the coordinating node in the distributed system from the data query request; for example, it can be extracted from a data query request that specifies a query time range as the specific data range information.

[0072] In step 202, the index metadata of the indexes in the index group can be obtained.

[0073] As mentioned above, in addition to the specific data range information included in the user's data query request, it is also necessary to obtain various data range information from the index group. As shown in step 302, the coordinating node can obtain the index metadata of the indexes in the index group. Optionally, in each coordinating node of the distributed system, for each read-only index in the index group, the data range information of the specific field name information in each shard of that read-only index can be obtained, and the metadata information including the data range information of the specific field name information of the read-only index can be cached in the memory of the coordinating node. For example, after the indexes of the index group IndexGroupX have undergone several rollovers, except for the latest index IndexGroupX-m which is currently receiving data writes, the data range information of the specific field name information of all the old indexes in the index group has been cached in each query coordinating node:

[0074] IndexGroupX-1: [range-1-start, range-1-end]

[0075] IndexGroupX-2: [range-2-start, range-2-end]

[0076] ...

[0077] IndexGroupX-m: None

[0078] Specifically, for any read-only index within the index group IndexGroupX, such as index IndexGroupX-2, its cached data range information can be represented as the data range range [range-2-start, range-2-end], where range-2-start and range-2-end represent the minimum and maximum values ​​of the data range, respectively, such as the minimum and maximum timestamps. However, for the index IndexGroupX-m, which is currently receiving data writes, its corresponding data range information is not cached on the coordinating node side because the data range information for specific field names is still changing.

[0079] Since the data range information of the index cached at the coordinating node will change every time the index rolls (i.e., when a write index switch event occurs) as the previous latest index changes to a read-only index, the data range information of the cached index needs to be updated when an index roll occurs.

[0080] Figure 4 This is a schematic flowchart illustrating the process of obtaining index metadata of an index in an index group according to an embodiment of the present disclosure.

[0081] According to embodiments of this disclosure, obtaining the index metadata of the indexes in the index group may include: updating the index metadata of all read-only indexes in the index group in response to a write index switching event occurring in the index group, wherein the update is used to ensure that the index metadata of all read-only indexes in the index group includes corresponding data range information for index filtering.

[0082] like Figure 4 As shown, the cluster's master node can be used to detect write index switching events in the index group. In response to the occurrence of a write index switching event (such as... Figure 4 As shown in S401, the master node can update the index metadata of all read-only indexes in the index group at each coordinating node.

[0083] According to embodiments of this disclosure, updating the index metadata of all read-only indexes in the index group may include: for each read-only index in the index group, checking whether the index metadata of the read-only index includes corresponding data range information; and if the index metadata of the read-only index does not include the corresponding data range information, obtaining the data range information of the read-only index and caching it in the index metadata of the read-only index. Figure 4 As shown, for each read-only index in the index group, you can check whether the corresponding data range information has been set in the index metadata of the read-only index (e.g., ...). Figure 4 As shown in S402 and S403, if the settings are confirmed, the index metadata can be retrieved and the subsequent query range filtering process can begin. However, if the settings are not confirmed, it is necessary to retrieve and set the data range information for the unset read-only index. Retrieving the data range information may involve obtaining the corresponding data range information from the data nodes containing each shard in the read-only index.

[0084] According to embodiments of this disclosure, when the master node detects a write index switching event in the index group, the master node can check whether the index metadata of each read-only index in the index group includes corresponding data range information. If the index metadata of the read-only index does not include the corresponding data range information, the master node can send a data range query request (e.g., ...) to the data node where the shard of the read-only index is located. Figure 4 As shown in S404), the data range information of the read-only index is obtained and cached in the index metadata of the read-only index, and the index metadata of the read-only index is synchronized to the coordinating node. Figure 4As shown, after receiving a data range query request from the master node, each slave node (data node) can query the data range information of its local shard (e.g., Figure 4 (As shown in S405), and returns the data range information query results to the master node for information updating. Next, the master node can update the index metadata (i.e., its data range information, such as...) of the corresponding read-only index at the master node based on the data range information query results obtained from each slave node (data node). Figure 4 As shown in S406), after the update is completed at the master node, the updated index metadata is distributed to each slave node (e.g., ...). Figure 4 As shown in S407), to update the index metadata at each slave node (such as...). Figure 4 As shown in S408, this enables data synchronization of index metadata.

[0085] Next, data queries can be performed based on the updated index metadata at the coordinating node. Figure 5 This is a timing diagram illustrating the entire data query process according to an embodiment of the present disclosure.

[0086] like Figure 5 As shown, the data query process of this disclosure may include three stages: an index filtering stage, a shard query stage, and a document crawling stage. The query range filtering of this disclosure may involve both the index filtering stage and the shard query stage. Optionally, in a distributed system, the entire data query process can be scheduled by a coordinating node.

[0087] During the index filtering stage, as shown in step 203, the index group can be filtered based on the acquired index metadata to determine a subset index group. The index metadata of each index in the subset index group includes data range information that overlaps with the specific data range information.

[0088] Optionally, during the index filtering stage, the obtained index metadata can be traversed first for index filtering (e.g., ...). Figure 5 As shown in S501, indexes irrelevant to the data query request (furthermore, the specific data range information of the data query request) are filtered out from the index group, while indexes related to the data query request can form a subset of index groups for subsequent sharding query stages. In this disclosure, an index related to the data query request can be considered an index whose data range information intersects with the specific data range information of the data query request; for example, the timestamp range corresponding to the index intersects with the query time range specified in the data query request.

[0089] Figure 6This is a schematic flowchart illustrating the index filtering stage in a data query process according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, index filtering of the index group based on acquired index metadata to determine a subset of the index group may include: determining one or more indexes in the subset index group that have data interval range information overlapping with the specific data interval range information based on a comparison between the specific data interval range information in the data query request and the data interval range information in the acquired index metadata, wherein the one or more indexes constitute the subset index group of the index group.

[0090] like Figure 6 As shown, query conditions can first be generated by parsing the query statement. For example, when a user enters the query statement "Find all data records from 10:00 to 11:00 on September 15, 2022" through a user terminal, the key information "10:00 to 11:00 on September 15, 2022" can be extracted to generate query conditions corresponding to that query statement (such as...). Figure 6 As shown in S601), for example, the query condition can be a specific data range information corresponding to the key information (e.g., a query time range, which can be represented by a timestamp range).

[0091] Next, you can iterate through all the indexes in the index group (such as...) Figure 6 As shown in S602 of the diagram, index filtering processing is performed for each index in the index group. This index filtering processing for each index may include index metadata acquisition, query condition comparison, and irrelevant index removal. According to embodiments of this disclosure, each index in the subset index group, along with its each shard and each data record segment, has corresponding data range information. For example... Figure 6 As shown, you can obtain the index metadata of this index (such as...). Figure 6 As shown in S603), and determine whether there is data range information in its index metadata (such as...). Figure 6 As shown in S604), if the index does not exist, it is directly removed from the index group and enters the index filtering process of the next index. However, if the index metadata contains data range information, the data range information can be compared with the query conditions (e.g., ...). Figure 6 As shown in S605). By determining whether the data range information overlaps with the query conditions (such as...). Figure 6As shown in S606, when the data range information overlaps with the query condition (i.e., there is overlap between the data range information and the specific data range information), it can be determined that the index is related to the data query condition, and therefore it can be retained for subsequent sharded queries. Conversely, when the data range information does not overlap with the query condition, it can be determined that the index is unrelated to the data query condition, and therefore the index can be removed from the index group (i.e., the subset index group) used for subsequent sharded queries (e.g., ...). Figure 6 As shown in S607), that is, subsequent shard queries will not query the shards of this index. Next, we can determine whether this index is the last index in the index group (e.g., Figure 6 As shown in S608, this determines whether the index filtering process for all indexes in the index group is complete. Therefore, by performing the above index filtering process on all indexes in the index group, the subset of indexes used for subsequent sharded queries can be determined.

[0092] In the index filtering stage described above, the data range information on the target index is compared with the query conditions (i.e., the characteristic data range information of the data query conditions) to determine whether the index and the query conditions intersect. If there is no intersection, the index is removed from the target index list, and then the subsequent sharding query stage begins. For example... Figure 5 As shown in step 303, this index filtering stage can be performed at the coordinating node. That is, the coordinating node can perform index filtering on the index group based on the acquired index metadata to determine a subset of the index group. The index metadata of each index in the subset index group includes data range information that overlaps with the specific data range information. Because of the pre-filtering (i.e., index filtering) by the coordinating node, only indexes that intersect with the query conditions remain in the subset index group, reducing the number of network requests sent by the coordinating node and the number of sharded queries by the data nodes, ultimately significantly reducing query time.

[0093] Next, we can proceed to the next processing stage (such as...) Figure 5 As shown in S502, a sharded query is performed based on the pre-filtered index group (i.e., the subset index group) for subsequent document crawling. In step 204, a sharded query can be performed on each index in the subset index group to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request.

[0094] According to embodiments of this disclosure, performing a sharded query for each index in the subset index group may include: for each index in the subset index group, performing shard filtering on all shards in the index based on the data range information of each shard in the index to determine a shard subset of the index, wherein the data range information of each shard in the shard subset includes at least a portion of the specific data range information; for each shard in the shard subset of the index, performing data record segment filtering on all data record segments in the shard based on the data range information of each data record segment in the shard to determine a data record segment subset of the shard, wherein the data range information of each data record segment in the data record segment subset includes at least a portion of the specific data range information; and determining multiple data records in all data record segment subsets corresponding to the specific data range information based on all data record segment subsets of all indexes in the subset index group.

[0095] Optionally, during the sharded query phase, shard filtering and data record segment filtering can be performed to filter out the smallest set of data record segments related to the data query request, and then search for the data record corresponding to the data query request in this set of data record segments. According to embodiments of this disclosure, the shard filtering of all shards in the index and the data record segment filtering of all data record segments in the shards are respectively based on the comparison of the data interval range information of all shards and the data interval range information of all data record segments with the specific data interval range information.

[0096] Optionally, similar to the index filtering described above, shard filtering can be performed based on a comparison of the query conditions with the data range information of all shards in each index within the subset index group to determine the shard subset of each index. Next, data record segment filtering can be performed based on a comparison of the query conditions with the data range information of all data record segments in each shard within each index's shard subset to determine the data record segment subset of each index. For example, the query conditions can be compared with the data range information on each index's shards. If there is no intersection, an empty result is returned (i.e., the shard is filtered out). If there is an intersection, the list of data record segments within that shard is traversed. Since each data record segment generates data range information and stores it in the data record segment's metadata during creation, when traversing the data record segments, the query conditions can be compared with the data range information of the data record segments. Similarly, if there is no intersection, the data record segment is filtered out; if there is an intersection, the data within the data record segment is compared to obtain a list of matching document IDs.

[0097] According to an embodiment of this disclosure, as shown in step 304, the coordinating node can send a shard query request to the data node where the shard of each index in the subset index group is located, and the data node performs a shard query based on the shard query request to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request.

[0098] For example, for an index that meets the query criteria, the coordinating node can send a shard query request to the data node where the index's shards reside (e.g., ...). Figure 5 As shown in S503). At the data node, the data range information for each shard (e.g., shard IndexGroupX-1-Shard1) can be represented as follows:

[0099] IndexGroupX-1-Shard1: [range-start, range-end]

[0100] The range-start and range-end represent the minimum and maximum values ​​of the data range for this partition, respectively, such as the minimum timestamp and the maximum timestamp.

[0101] At the data node, the data range information for each data record segment within the fragment can be represented as follows:

[0102] IndexGroupX-1-Shard1-Segment1: [range-1-start, range-1-end]

[0103] IndexGroupX-1-Shard1-Segment2: [range-2-start, range-2-end]

[0104] ...

[0105] IndexGroupX-1-Shard1-Segmentn: [range-n-start, range-n-end]

[0106] For any data record segment, such as IndexGroupX-1-Shard1-Segment2, range-2-start and range-2-end represent the minimum and maximum values ​​of the data range of that data record segment, such as the minimum and maximum timestamps.

[0107] Therefore, in a distributed system, after receiving a shard query request for any shard (e.g., IndexGroupX-1-Shard1), a data node can execute the shard query locally (e.g., ...). Figure 5 As shown in S504, this involves first extracting the specific data range information from the shard query request and comparing it with the data range information of IndexGroupX-1-Shard1 cached in memory. This allows for a quick determination of whether the data ranges overlap. If they do not overlap, the shard is skipped (i.e., filtered out). If they overlap, each data record segment within the shard is traversed to determine if its data range overlaps. Similarly, if they do not overlap, the data record segment is quickly skipped, thus quickly pruning irrelevant data record segments. The determined subset of corresponding data record segments is then returned to the coordinating node (e.g., ...) as the shard query result. Figure 5 (As shown in S505).

[0108] Therefore, by using query range filtering processes such as index filtering, shard filtering, and data record segment filtering, query requests can be avoided during the query phase by sending query requests to a large number of index shards with no data overlap, thereby greatly reducing query latency and cluster invalid query load and improving user experience.

[0109] After completing the sharded query, the document crawling phase can proceed (e.g., ...). Figure 5 As shown in S506, this is to fetch data records corresponding to the data query request from the data record segment subset of each index in the subset index group. For example, the coordinating node can send a document fetch request to the corresponding data node based on a list of document IDs (such as...). Figure 5 As shown in S507), this allows data nodes to perform document fetching within the data node based on a list of matched document IDs (e.g., ...). Figure 5 As shown in S508, the obtained data records are returned to the coordinating node (e.g., ...). Figure 5 As shown in S509, this is then output to the user terminal.

[0110] The data query method disclosed herein can be applied to various scenarios requiring data querying. For example, this data query method can be applied to a highly available, scalable, fully managed cloud-based Elasticsearch service based on the open-source search engine Elasticsearch, and users can specify options to enable the query range filtering function of the coordinating node when performing data queries.

[0111] For example, the query range filtering for the coordinating node can be enabled by default (disabled by default) through dynamic cluster settings, as shown in the following example:

[0112] PUT _cluster / settings

[0113] {

[0114] "transient": {

[0115] "action.search.pre_filter.enabled": "true"

[0116] },

[0117] "persistent": {

[0118] "action.search.pre_filter.enabled": "true"

[0119] }

[0120] }

[0121] Specifically, when the identifier "action.search.pre_filter.enabled" for query range filtering is set to true, all queries will undergo query range filtering by the coordinating node; otherwise, it will not. This selective setting allows users to customize their desired data query methods.

[0122] Furthermore, when the data query method described above is applied to time-series scenarios to query recent data, the number of shards that intersect with the query conditions accounts for only a small fraction of the total number of shards, and the user's query time is significantly reduced. Specifically, the optimization effect on query time is negatively correlated with the proportion of shards that intersect with the query data; the lower the proportion of relevant data, the more significant the optimization effect on query time. Therefore, the data query method disclosed above can achieve significant query performance optimization in time-series scenarios.

[0123] Figure 7 This is a schematic diagram illustrating a data query device 700 according to an embodiment of the present disclosure.

[0124] According to embodiments of this disclosure, the data query device 700 may include a request receiving module 701, a data acquisition module 702, an index filtering module 703, and a sharded query module 704.

[0125] The request receiving module 701 can be configured to receive data query requests for an index group, the data query requests including specific data range information. Optionally, the request receiving module 701 can perform the operations described above with reference to steps 201 and 301.

[0126] Optionally, the index group of this disclosure may include multiple indexes, each of which may store related data. For example, in scenarios such as log recording, indexes are typically rolled over daily, so each index may correspond to a data set in units of one day (or other time metrics), and the index group may include multiple indexes corresponding to multiple consecutive time periods. Optionally, the data in each index may be distributed to different nodes in a distributed system for storage; that is, the data in each index may be distributed across its multiple shards, which may be located on different machines (i.e., different data nodes in a distributed network) to achieve horizontal scaling of data in the distributed system. By sharding the data in the index, parallel operations can be performed during data writing or querying, i.e., reading and writing data on various data nodes, thereby improving throughput. Furthermore, in each shard, data may be stored in the form of data record segments, each of which is an independent data storage unit that supports any query by the user. It may be a set of immutable indexes and data files generated for data records (documents) produced for a certain continuous time window.

[0127] Optionally, since index rolling may occur within the index group—that is, a new index is created and data writes are transferred to the newly created index—this newly created index can be responsible for receiving new data writes, while all previous indexes will no longer receive new data writes and will be marked as "read-only" indexes. In the index group, all indexes except the latest one currently receiving new data writes are read-only indexes. Therefore, since the data of read-only indexes is deterministic and immutable, their index metadata can be cached in each coordinating node of the distributed system for use in subsequent data queries.

[0128] Optionally, the metadata of each index, shard, and data record segment can include data range information for a specific field name. This specific field name information can be a predefined field set for query range filtering in this disclosure (this predefined field can be specified by the user during index group creation via configuration), and the data range information can correspond to statistical information for that specific field name information. For example, for a data query request based on a time range, the specific field name information can be the timestamp information of the data (such as the timestamp of each data record), and the data range information can be a statistical representation of the timestamp information of the data.

[0129] As described above, in the embodiments of this disclosure, query range filtering can be performed based on the specific data range information included in the user's data query request and the various data range information in the index group, thereby improving query performance.

[0130] As described above, in addition to the specific data range information included in the user's data query request, it is also necessary to obtain various types of data range information from the index group. Therefore, the data acquisition module 702 can be configured to acquire the index metadata of the indexes in the index group. Optionally, the data acquisition module 702 can perform the operations described above with reference to steps 202 and 302. As an example, in each coordinating node of the distributed system, for each read-only index in the index group, the data range information of the specific field name information in each shard of the read-only index can be acquired, and the metadata information including the data range information of the read-only index regarding the specific field name information can be cached in the memory of the coordinating node.

[0131] Because the data range information of the indexes cached at the coordinating node changes with each index rollover (i.e., when a write index switch event occurs) as the previous latest index becomes a read-only index, the cached data range information needs to be updated during index rollover. For example, for each read-only index in an index group, it's possible to check if the corresponding data range information is set in the index metadata. If it's determined that it's set, the index metadata can be retrieved and subsequent query range filtering can proceed. If it's determined that it's not set, then the data range information needs to be retrieved and set for the read-only indexes that haven't been set. Retrieving the data range information can include obtaining the corresponding data range information from the data nodes where each shard in the read-only index resides.

[0132] Next, data queries can be performed based on the updated index metadata at the coordinating node, which can be implemented by the index filtering module 703 and the shard query module 704. Optionally, the data query process of this disclosure may include three stages: an index filtering stage, a shard query stage, and a document fetching stage, wherein the query range filtering of this disclosure may involve the index filtering stage and the shard query stage. Optionally, in a distributed system, the entire data query process can be scheduled by the coordinating node.

[0133] The index filtering module 703 can be configured to perform index filtering on the index group based on the acquired index metadata to determine a subset of the index group, wherein the index metadata of each index in the subset of the index group includes data range information that overlaps with the specific data range information. Optionally, the index filtering module 703 can perform the operations described above with reference to steps 203 and 303.

[0134] Optionally, during the index filtering stage, the acquired index metadata can be traversed first to perform index filtering, thereby removing indexes from the index group that are irrelevant to the data query request (further, the specific data range information of the data query request). The indexes related to the data query request can form a subset of index groups for subsequent sharding query stages. In this disclosure, an index related to the data query request can be considered an index whose data range information intersects with the specific data range information of the data query request; for example, the timestamp range corresponding to the index intersects with the query time range specified in the data query request.

[0135] In the index filtering module 703, the data range information on the target index is compared with the query conditions (i.e., the characteristic data range information of the data query conditions) to determine whether the index and the query conditions have any intersection. If there is no intersection, the index is removed from the target index list and then the sharding query stage is carried out in the sharding query module 704.

[0136] The shard query module 704 can be configured to perform a shard query for each index in the subset index group to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request. Optionally, the shard query module 704 can perform the operations described above with reference to steps 204 and 304.

[0137] Optionally, in the sharded query module 704, sharding filtering and data record segment filtering can be performed to filter out the smallest set of data record segments related to the data query request, and then search for the data record corresponding to the data query request in the set of data record segments.

[0138] Optionally, similar to the index filtering process in index filtering module 703, shard filtering can be performed based on a comparison of the query conditions with the data range information of all shards in each index within the subset index group to determine the shard subset of each filtered index. Next, data record segment filtering can be performed based on a comparison of the query conditions with the data range information of all data record segments in each shard within the shard subset of each index to determine the data record segment subset of each filtered index. For example, the query conditions can be compared with the data range information on each index's shards. If there is no intersection, an empty result is returned directly (i.e., the shard is filtered out). If there is an intersection, the list of data record segments within that shard is traversed. Since each data record segment generates data range information and stores it in the data record segment's metadata during creation, when traversing the data record segments, the query conditions can be compared with the data range information of the data record segments. Similarly, if there is no intersection, the data record segment is filtered out; if there is an intersection, the data within the data record segment is compared to obtain a list of matching document IDs. Therefore, the data records corresponding to the data query request can finally be crawled from the data record segment subset of each index in the subset index group. For example, document crawling can be performed in the data node based on the matching list of document IDs to return the obtained data records to the user terminal.

[0139] Therefore, by using the query interval filtering processes such as index filtering, shard filtering and data record segment filtering in the data query apparatus 700 of this disclosure, query requests can be avoided from being sent to a large number of index shards with no data overlap during the query phase, thereby greatly reducing query latency and cluster invalid query load and improving user experience.

[0140] According to another aspect of this disclosure, a data query device is also provided. Figure 8 A schematic diagram of a data query device 2000 according to an embodiment of the present disclosure is shown.

[0141] like Figure 8 As shown, the data query device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code, which, when executed by the one or more processors 2010, can perform the data query method described above.

[0142] The processor in the embodiments of this disclosure can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0143] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0144] For example, the method or apparatus according to embodiments of this disclosure can also be used by means of Figure 9 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the data query method provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 8 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 9 One or more components in the computing device shown.

[0145] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 10 A schematic diagram 4000 of a storage medium according to the present disclosure is shown.

[0146] like Figure 10As shown, the computer storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, a data query method according to an embodiment of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory used in the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0147] Embodiments of this disclosure also provide a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a data query method according to embodiments of this disclosure.

[0148] The embodiments of this disclosure provide a data query method, a data query method for a distributed system, and corresponding apparatus, devices, and storage media.

[0149] Compared to existing data query methods, the method provided by the embodiments of this disclosure can find partial indexes and partial fragments associated with user data requests from a large number of indexes, thereby improving the efficiency of data query and reducing the time consumption of data query.

[0150] The method provided in the embodiments of this disclosure performs index filtering based on specific data range information in the data query request and data range information of the indexes in the index group. This removes indexes in the index group that are irrelevant to the data query request. Then, it searches for data records corresponding to the specific data range information in the data query request from the filtered index group through sharded queries, thereby achieving efficient data querying. The method in the embodiments of this disclosure avoids sending query requests to shards of a large number of indexes with no data overlap during the query phase by filtering query ranges, thus greatly reducing query latency and cluster invalid query load, and improving user experience.

[0151] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0152] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0153] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations can be made to these embodiments or their features without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure.

Claims

1. A data query method, comprising: Receive a data query request for an index group, the data query request including specific data range information; Obtain the index metadata of the indexes in the index group, wherein the data range information in the index metadata of each index is dynamically aggregated based on the data range information of its lower-level data units; Based on the acquired index metadata, index filtering is performed on the index group to determine a subset of the index group. The index metadata of each index in the subset of the index group includes data range information that overlaps with the specific data range information. For each index in the subset index group, perform a shard query to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request; Each index in the index group includes one or more shards, each shard includes one or more data record segments, each data record segment includes one or more data records, and each index, its shards, and each data record segment have corresponding data range information. The data range information in the index metadata of the index, which is dynamically aggregated based on the data range information of the lower-level data units of each index, includes: For each data record segment in each shard of the index, the data range information of the data record segment is statistically obtained based on the specific field name information of all data records included in the data record segment; For each shard in the index, the data interval range information of the shard is determined by aggregation based on the data interval range information of all data record segments included in the shard; and For the index, the data range information in the index metadata is determined by aggregating the data range information of all shards included in the index.

2. The method as described in claim 1, wherein, The index group includes a latest index and a read-only index. The latest index is the latest index that is currently receiving new data writes, and the read-only indexes are the other indexes in the index group besides the latest index. In this system, the read-only index no longer receives new data writes, and the index metadata of the read-only index is cached for data querying. The cached index metadata of the read-only index includes the data range information of the read-only index. The data range information of the latest index is not cached for data querying before the next write index switching event occurs in the index group. The latest index is changed to a read-only index in response to the next write index switching event.

3. The method as described in claim 2, wherein, Retrieving the index metadata of the indexes in the index group includes: In response to a write index switching event occurring in the index group, the index metadata of all read-only indexes in the index group is updated, wherein the update is used to ensure that the index metadata of all read-only indexes in the index group includes corresponding data range information for index filtering.

4. The method of claim 3, wherein, Updating the index metadata for all read-only indexes in the index group includes: For each read-only index in the index group, check whether the index metadata of the read-only index includes the corresponding data range information; and If the corresponding data range information is not included in the index metadata of the read-only index, the data range information of the read-only index is obtained and cached in the index metadata of the read-only index.

5. The method of claim 1, wherein, Based on the acquired index metadata, the index group is filtered to determine a subset of the index group, including: Based on the comparison between the specific data range information in the data query request and the data range information in the obtained index metadata, one or more indexes in the subset index group that have data range information that overlaps with the specific data range information are determined, and the one or more indexes constitute the subset index group of the index group.

6. The method of claim 1, wherein, The data range information is indicated by the specific field name information, which is related to the data query request. The data range information for each data record segment is statistically calculated synchronously based on the specific field name information of the data record segment during the generation of the data record segment.

7. The method of claim 6, wherein, For each index in the subset index group, sharded queries include: For each index in the subset index group, based on the data range information of each shard in the index, all shards in the index are filtered to determine the shard subset of the index, wherein the data range information of each shard in the shard subset includes at least a portion of the specific data range information; For each shard in the shard subset of the index, based on the data interval range information of each data record segment in the shard, data record segment filtering is performed on all data record segments in the shard to determine a subset of data record segments in the shard. The data interval range information of each data record segment in the subset of data record segments includes at least a portion of the specific data interval range information; and Based on all data record segments of all indexes in the subset index group, determine multiple data records in the subset of all data record segments that correspond to the specific data range information.

8. The method of claim 7, wherein, The shard filtering of all shards in the index and the data record segment filtering of all data record segments in the shards are respectively based on the comparison of the data interval range information of all shards and the data interval range information of all data record segments with the specific data interval range information.

9. A data query method for a distributed system, the distributed system including a coordinating node and data nodes, the method comprising: The coordinating node receives a data query request for an index group, the data query request including specific data range information; The coordinating node obtains the index metadata of the indexes in the index group, wherein the data range information in the index metadata of each index is dynamically aggregated based on the data range information of its lower-level data units; The coordinating node performs index filtering on the index group based on the acquired index metadata to determine a subset of the index group. The index metadata of each index in the subset of the index group includes data range information that overlaps with the specific data range information. The coordinating node sends a shard query request to the data node where the shard of each index in the subset index group is located, and the data node performs a shard query based on the shard query request to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request; Each index in the index group includes one or more shards, each shard includes one or more data record segments, each data record segment includes one or more data records, and each index, its shards, and each data record segment have corresponding data range information. The data range information in the index metadata of the index, which is dynamically aggregated based on the data range information of the lower-level data units of each index, includes: For each data record segment in each shard of the index, the data range information of the data record segment is statistically obtained based on the specific field name information of all data records included in the data record segment; For each shard in the index, the data interval range information of the shard is determined by aggregation based on the data interval range information of all data record segments included in the shard; and For the index, the data range information in the index metadata is determined by aggregating the data range information of all shards included in the index.

10. The method of claim 9, wherein, The distributed system also includes a master node, which is used to detect the occurrence of write index switching events in the index group; Specifically, when the master node detects a write index switching event in the index group, the master node checks whether the index metadata of each read-only index in the index group includes the corresponding data range information; and If the corresponding data range information is not included in the index metadata of the read-only index, the master node sends a data range query request to the data node where the shard of the read-only index is located to obtain the data range information of the read-only index and cache it in the index metadata of the read-only index, and synchronizes the index metadata of the read-only index to the coordinating node.

11. A data query device, comprising: The request receiving module is configured to receive data query requests for an index group, the data query requests including specific data range information; The data acquisition module is configured to acquire the index metadata of the indexes in the index group, wherein the data range information in the index metadata of each acquired index is dynamically aggregated based on the data range information of its lower-level data units; An index filtering module is configured to perform index filtering on the index group based on the acquired index metadata to determine a subset of the index group, wherein the index metadata of each index in the subset of the index group includes data range information that overlaps with the specific data range information; and The sharded query module is configured to perform sharded queries on each index in the subset index group to return multiple data records corresponding to the specific data range information in all shards of all indexes in the subset index group, as the query response result of the data query request; Each index in the index group includes one or more shards, each shard includes one or more data record segments, each data record segment includes one or more data records, and each index, its shards, and each data record segment have corresponding data range information. The data range information in the index metadata of the index, which is dynamically aggregated based on the data range information of the lower-level data units of each index, includes: For each data record segment in each shard of the index, the data range information of the data record segment is statistically obtained based on the specific field name information of all data records included in the data record segment; For each shard in the index, the data interval range information of the shard is determined by aggregation based on the data interval range information of all data record segments included in the shard; and For the index, the data range information in the index metadata is determined by aggregating the data range information of all shards included in the index.

12. The apparatus of claim 11, wherein, Based on the acquired index metadata, the index group is filtered to determine a subset of the index group, including: Based on the comparison between the specific data range information in the data query request and the data range information in the obtained index metadata, one or more indexes in the subset index group that have data range information that overlaps with the specific data range information are determined, and the one or more indexes constitute the subset index group of the index group.

13. A data query device, comprising: One or more processors; as well as One or more memories storing a computer-executable program, which, when executed by the processor, performs the method of any one of claims 1-10.

14. A computer program product stored on a computer-readable storage medium and comprising computer instructions that, when executed by a processor, cause a computer device to perform the method of any one of claims 1-10.

15. A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

Images